Anubhav Gain

Security Software Engineer · XDR/OXDR Architect · Rust & eBPF · 222+ Research Citations

Tags

/ 2fa ab aboutme access-control access-management active-directory actix-web ad-blocking administration admission-control adr advanced-devops advanced-threats agile ai AI AI Agents AI Development ai-cybersecurity ai-detection ai-integration ai-powered-scoring AI-security ai-threat-detection ai-threat-hunting ai-translation airtable alert-consolidation alert-fatigue alerting amd-sev amplitude amtd analysis analytics Android anomaly-detection ansible api API api-client api-design api-gateway api-management api-security app-control-for-business apparmor apple Apple-Intelligence Apple-security applescript application-security applications applocker APT-attacks apt-detection arch-linux architecture Architecture architecture-patterns argocd arm-trustzone asgi assemblyscript astro-ai async Async athena Attack-Mitigation attestation audio auth0 authentication Authentication authorization automated-monitoring automated-response automation autoscaling aws AWS awslambda aya Aya azure azure-ad backend background-services backstage backup Backup bare-metal baseline-analysis bash basics bcc behavioral-analysis behavioral-analytics benchmarking Best Practices best-practices bgp bind-mounts biometric-security blacklist blockchain blog-platforms Blogging blue-green blue-team bluechi bochs borrowing bot-management Bot-Management boto3 bpftrace broadcom browser build build-configuration busybox byzantine c caddy calico canary career Career cdc cdn CDN centos certificate-authority certificates chartmuseum chatbot chatops check_wmi_plus choreography chrome ci-cd CI/CD cicd cilium circuit-breaker cis cis-benchmark cka Claude claude-3-opus claude-code cleanup cli cli-tools clickhouse cloud cloud-native cloud-providers cloud-security Cloud-Storage cloudflare Cloudflare cloudflared cloudrun cluster cluster-deployment cluster-health cluster-management cluster-setup cmd cni cocktails code management code-generation collaboration communication communication-patterns community compensation compliance compliance-automation compliance-reporting Compose compression conference-translation confidential-computing configmaps configuration configuration-management consensus consul container Container Orchestration container-management container-monitoring container-orchestration container-runtime container-security containerization Containerization containers Containers content-automation content-generation contextual-translation continuous-improvement contract-testing controller controller-manager cookiecutter cooking coredns coreos Coroutines correlation correlation-rules cors cosmopolitan Cost Optimization cost-optimization Cost-Optimization cpp cqrs crd cri-o cronjob cronjobs cross-account cross-cloud Cross-Platform cryptography csharp css custom linux custom-decoders customization CVE-2025-31200 CVE-2025-31201 cybersec cybersecurity Cybersecurity d1-database D1-Database daemonsets dashboard dashboards data-analytics data-architecture data-channels data-consistency data-fetcher data-governance data-management data-masking data-migration data-pipelines data-prepper data-processing data-protection data-recovery database DataBinding datasette dba ddos-protection DDoS-Protection debian debugging decoders deepl-voice deepseek-r1 defense defensive-security Demo deno dep deployment Deployment design-patterns desktop-development detection detection-accuracy Developer Tools developer-portal developer-portals development devops DevOps devops-culture devops-journey devsecops DevSecOps devtools devtron diagrams digitalocean disaster-recovery discord discovery disk-encryption disk-provisioning Distributed Systems distributed-security distributed-systems distributed-transactions distro distrobox django dkim dmarc dnf dns docker Docker document-processing documentation domain-administration domain-driven-design dotnet duckdb dx-operational-observability dynamodb ebpf eBPF ec2 ECS edge-computing Edge-Database edge-devices edge-functions edge-security edr elastic-alternative elasticsearch electron elk-stack email email-automation embedded linux embedded-systems encryption endpoint-protection endpoint-security Engineering enterprise Enterprise Enterprise Security enterprise-architecture enterprise-clustering enterprise-governance enterprise-integration enterprise-security environment-variables envoy error-handling etcd ETL eureka event-driven event-driven-architecture event-sourcing event-streaming Example exif exploit-mitigation exploit-prevention exploitation falco Fargate fault-tolerance feature-flags federation fedora fedora-coreos ffmpeg FIDO2 file-integrity file-rule-levels file-transfer filebeat FileVault fips fips-203 firefox firewall Flow fluentbit flux fly forensics full-stack functions fundamentals future-translation gainsaheb Gatekeeper gcp gcs gemini-2.5 general gis git github github-actions gitlab gitops GitOps Global-Distribution gmail go golang google google-authenticator google-cloud google-sheets googlecloud governance gpt gpt-4o gpt3 grafana graph-api graphical interface graphql GraphQL group-policy grpc gui guide hacker-news ham-radio hardening hardware hardware-acceleration hardware-security hashicorp health-probes helm helm-charts heroku high-availability high-risk-security Hilt hirte history homebrew homelab hpa hsm html http http3 https httpx hugo hybrid-cloud hybrid-quantum-classical hypothesis-driven iac iam icinga ics identity identity management identity-governance identity-management ignition imagemagick incident-response index index-management indexer industrial-iot industrial-security Infopercept infrastructure Infrastructure infrastructure-as-code ingress insider-threats installation instrumentation integration Integration Testing integration-testing intel-sgx internet interoperability introduction intrusion-detection inventory Invinsense ios iOS-development iOS-security iot isa istio iterators jamstack jasmin java javascript JavaScript jenkins jest Jetpack Jetpack Compose jinja jq json jsonpath jupyter jwt JWT k8s kafka kannel kaslr keepalive kernel Kernel kernel-security key-management keycloak KIND kiota kms Kotlin kprobe kpti kubeadm kubectl kubernetes Kubernetes kubernetes-security kustomize kyber labels lambda language-processing large-language-models lattice-cryptography launchd learning Legacy Code legacy-systems libvirt lightsail lightweight distro linkding linkerd linux Linux linux development linux from scratch linux kernel linux kernel compilation linux system linux-kernel linux-security LiveData liveness lkl llm LLM llm-translation llms load-testing Lockdown-Mode log log-analysis log-ingestion log-management log-parsing logging logs low-latency lsm Machine Learning machine-learning machine-translation machinelearning macos macOS macOS-development macOS-security malware malware-analysis malware-detection malware-protection management manifest maps Markdown markdown master-keys mastodon mdm Media-Storage mediawiki Memory Management memory-management memory-safety mermaid message-queue messaging metrics metrics-server MFA mfa micro-segmentation micromdm microservices Microservices microsoft microsoft-copilot microsoft-graph microsoft-kiota microwindows midjourney migration Migration Migrations minimalistic os minio misc mitigation mitre-attack ml-integration ml-kem mobile-device-management mobile-security monitoring Monitoring morphisec mtls Multi-Agent Systems multi-cloud multi-cluster multi-tenancy multi-tenant multilingual-blogs multimodal-ai multipass musl MVVM n8n nagios Namespaces nano-x nats Navigation netdata netflix Network network-access network-correlation network-security networking Networking neural-machine-translation neural-networks neuvector nextjs-ai nfs nginx nist-standards no-code node node-affinity node-exporter Node.js nodejs noisy-neighbors nosql notifications npm NSO-group oauth oauth2 OAuth2 object-storage Object-Storage objective-c observability observable observable-plot oci-runtime ocr offensive-security oidc open source open-xdr openai openapi opensearch openssh OpenSSL openssl opentelemetry openvpn operating system operating-systems operators optimization Optimization oracle oracle23c orchestration organizational-charts ot-ics overture-maps owasp ownership OXDR P2P p2p package-management packaging packet-capture packet-processing pact pages pagination partitioning passkeys passwordless patterns pbft pdf peer-to-peer Pegasus-protection performance Performance performance-benchmarking performance-optimization permissions persistentvolumeclaims persistentvolumes personalization php pihole pipeline pixelmator pixie pkcs11 PKI pki Platform Development platform-engineering playwright pluggy plugin plugins pmp pod-security podman pods polyglot-persistence post-quantum-cryptography postgresql powershell presenting pricing priority-management privacy-controls privacy-engineering privacy-protection Private-Cloud-Compute process-exporter processor production Production production-deployment productivity programming project-management prometheus protocols proxy purpleair pyodide pypi pytest python qemu quadlet quadlets quality-assurance quantum-acceleration quantum-ai quantum-algorithms quantum-computing quantum-nlp quantum-resistance quantum-resistant quarto quic r2 R2 r2-storage rabbitmq raft ransomware-defense rate-limiting Rate-Limiting rbac RBAC rdp react readiness readthedocs real-time real-time-analytics real-time-translation red-team reddit Refactoring reference regulatory-compliance remote-access ReplicaSet repository management resilience resilience4j resource-management resources REST rest-api restore risc-v risk-based-alerting rocky-linux roles rolling-updates Room rootkit rootless rootless-containers routing rpki rpm-ostree rsyslog rule-engine rule-options rules runtime-protection runtime-security rust Rust s3 s3-compatible S3-Compatible safari safety-critical saga-pattern sandboxed-execution sandboxing scalability sched_ext scheduler scheduling scim screen-sharing sdk-development sdk-generation sdlc seamlessm4t search search-engine seccomp secrets secrets-management secure-boot secure-coding secure-element secure-enclave secure-enclaves security Security Security Platform security-analytics security-architecture security-automation security-commands security-implementation security-monitoring security-orchestration security-patches security-platform security-runtimes security-systems security-testing security-tools security-updates selenium selinux seo seo-optimization server-setup serverless Serverless service mesh service-accounts service-discovery service-mesh service-workers Services shell shell-configuration shell-scripting shellcode shot-scraper siem SIEM signaling sigstore simultaneous-interpretation single-node sinkhole site-speed slack slsa smack smallstep smpp SMS sms-gateway smtp snapshot soar software development software-attestations software-testing spatialite speech-translation spf sphinx spiffe spire spreadsheet spring-boot spyware-protection sql SQL sqlite SQLite squarespace sre ssh ssl SSL/TLS starship State Management StateFlow static-sites stepca storage storageclass streaming STUN supply-chain-security svg swift sysadmin sysmon System Architecture system services system-administration system-calls system-design system-extension system-integrity system-maintenance system-programming systemd systems-programming tailscale taints targeted-attacks TCC tcp team-collaboration telegram terminal terminal-services terraform tesseract testcontainers testing Testing tetragon textract threading threat-detection threat-hunting threat-intelligence threat-modeling threat-prevention threat-remediation threshold-cryptography tiktok tinyemu tls TLS toast-notifications token-flow tokio Tokio tolerations tomcat tools tpm tracee tracing Traffic-Analysis troubleshooting trusted-execution trusted-execution-environments trusted-publishing tunnel tunneling tunnels turing TURN tutorial twitter typescript ubuntu udp ui UI UI Testing uninstallation unisolation Unit Testing unix utilities upgrade uprobes usb-control use-cases user-experience user-monitoring user-session utm-stack valtown vault vega vendor-lock-in version control version-control Video video video-chat ViewModel vim violation-detection virtualization visualization vmware volumes vpn vpn-replacement vscode vulnerabilities vulnerability vulnerability-analysis vulnerability-detection vulnerability-scanning waf WAF wasi wasm wazuh Wazuh wdac web Web Development web-components web-development web-filtering web-infrastructure web-performance web-security Web-Security web-servers webassembly webauthn WebAuthn webhooks webrtc WebRTC websockets white-labeling wikipedia windows Windows Services windows-10-iot windows-api windows-monitoring windows-security winexe winrt wmi wordpress worker-nodes workers Workers workflow workflow-automation workload-identity xdp XDP xdr XDR xdr-platform xdr-testing xpc xprotect XProtect yaml youtube zeit-now zero-copy zero-day zero-trust zsh ztna

239 words

1 minute

Named Entity Resolution with dslim/distilbert-NER

Anubhav Gain

2024-10-29

llms

/

llms

/

python

/

javascript

/

web

/

cli

Named Entity Resolution with dslim/distilbert-NER#

I was exploring the original BERT model from 2018, which is mainly useful if you fine-tune a model on top of it for a specific task.

dslim/distilbert-NER by David S. Lim is a popular implementation of this, with around 20,000 downloads from Hugging Face every month.

I tried the demo from the README but it didn’t quite work - it complained about an incompatibility with Numpy 2.0.

So I used uv run --with 'numpy<2.0' to run it in a temporary virtual environment. Here’s a Bash one-liner that demonstrated the model:

1
uv run --with 'numpy<2.0' --with transformers python -c '
2
from transformers import AutoTokenizer, AutoModelForTokenClassification
3
from transformers import pipeline
4
import json
5
model = AutoModelForTokenClassification.from_pretrained("dslim/distilbert-NER")
6
tokenizer = AutoTokenizer.from_pretrained("dslim/distilbert-NER")
7
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
8
text = "This is an example sentence about Simon Willison who lives in Half Moon Bay"
9
print(json.dumps(nlp(text), indent=2, default=repr))'

The first time you run this it will download 250MB to your ~/.cache/huggingface/hub/models--dslim--distilbert-NER folder.

Example output:

1
[
2
  {
3
    "entity": "B-PER",
4
    "score": "0.9982101",
5
    "index": 7,
6
    "word": "Simon",
7
    "start": 34,
8
    "end": 39
9
  },
10
  {
11
    "entity": "I-PER",
12
    "score": "0.99835676",
13
    "index": 8,
14
    "word": "Willis",
15
    "start": 40,
16
    "end": 46
17
  },
18
  {
19
    "entity": "I-PER",
20
    "score": "0.9977602",
21
    "index": 9,
22
    "word": "##on",
23
    "start": 46,
24
    "end": 48
25
  },
26
  {
27
    "entity": "B-LOC",
28
    "score": "0.99432063",
29
    "index": 13,
30
    "word": "Half",
31
    "start": 62,
32
    "end": 66
33
  },
34
  {
35
    "entity": "I-LOC",
36
    "score": "0.99325883",
37
    "index": 14,
38
    "word": "Moon",
39
    "start": 67,
40
    "end": 71
41
  },
42
  {
43
    "entity": "I-LOC",
44
    "score": "0.9919292",
45
    "index": 15,
46
    "word": "Bay",
47
    "start": 72,
48
    "end": 75
49
  }
50
]