45 tok/s on a Laptop: Getting Qwen3.6-27B's NVFP4 Quants Running on llama.cpp

2026-07-18

Qwen3.6-27B is arguably the best model you can fit on a 24 GB GPU today. But “fit” and “fast” are different problems and the most interesting quantization format for Blackwell hardware, NVFP4, doesn’t run on the tools most of us actually use locally. This is the story of how we got it running at nearly 2x our starting throughput, including a detour into patching llama.cpp’s conversion script to handle a checkpoint it flatly refused to load.

TL;DR: We repackaged the NVIDIA and unsloth NVFP4 checkpoints of Qwen3.6-27B into GGUF, so they run on llama.cpp’s native sm_120 FP4 tensor-core path. On an RTX 5090 Laptop (24 GB), combined with the model’s MTP head for speculative decoding, this took us from 24 tok/s (a popular Q4_K_M build) to 45 tok/s at 160K context. The NVIDIA checkpoint converts out of the box; the unsloth one needed a small patch to the convert script, which we describe in detail.

The wen3.6 27B NVFP4 gguf repack performs around double the baseline of the unsloth Q4_K_M quant
The wen3.6 27B NVFP4 gguf repack performs around double the baseline of the unsloth Q4_K_M quant

Read More

Fixing Bluetooth On Pop!_OS And ASUS ProArt P16

2026-03-28

Fixing Mouse And Keyboard Stutter

The ASUS ProArt P16 seems to have a bug in the power management that makes bluetooth connections stutter when the Nvidia GPU wakes up: #3878. This is quite annoying but you can work around it by enabling the respective devices permanently:

echo on | sudo tee /sys/bus/pci/devices/0000:64:00.0/power/control
echo on | sudo tee /sys/bus/pci/devices/0000:64:00.1/power/control

Connecting a Keyboard With Pairing Key

Pop!_OS is currently lacking a way to pair device that require a passkey through the UI. However, you can use bluetoothctl to pair the device. In our example we use a Logitech MX Keys Mini keyboard. This specific device can be put into pairing mode by holding one of the three device selectors until it blinks rapidly.

Other devices may differ in this regard. Anyway, put it into pairing mode and open a terminal, then follow these steps:

# Open the bluetoothctl tool
bluetoothctl

# Enable the controller to search for your device
scan on
# List all discovered devices, note the MAC address of your device
devices

# To avoid problems, restart the correct agent
agent off
agent KeyboardOnly

# Now pair your device using the MAC address discovered before
pair D5:44:64:5F:DC:EA

# Set a passkey
[agent] Enter passkey (number in 0-999999): 123456
# Now repeat the passkey on the keyboard and your device should be connected

Read More

no-title

2025-12-29

Read More

Privacy First Summarizing with AI

2025-12-29

Motivation

Design Decisions

Algorithm

Alternatives

Future Work

Read More

A webpack hexo bundling pipeline setup

2025-12-29

Passing command line arguments to hexo templates

npx hexo generate –variable value

Read More

git sslv3 alert bad record mac

2025-12-29
benjamin@Asterix:blog# git push
Enumerating objects: 138, done.
Counting objects: 100% (138/138), done.
Delta compression using up to 16 threads
Compressing objects: 100% (118/118), done.
Writing objects: 100% (118/118), 45.47 MiB | 32.04 MiB/s, done.
Total 118 (delta 63), reused 0 (delta 0), pack-reused 0
error: RPC failed; curl 56 LibreSSL SSL_read: error:1404C3FC:SSL routines:ST_OK:sslv3 alert bad record mac, errno 0
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date
benjamin@Asterix:blog# git fetch
benjamin@Asterix:blog# GIT_CURL_VERBOSE=1 GIT_TRACE=1 git push
23:38:25.362521 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/bin/git
23:38:25.363107 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/bin
23:38:25.363867 git.c:444               trace: built-in: git push
23:38:25.364345 run-command.c:664       trace: run_command: GIT_DIR=.git git remote-https origin https://gitlab.fritz.box/GevatterGaul/blog.git
23:38:25.366739 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
23:38:25.367244 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.367810 git.c:730               trace: exec: git-remote-https origin https://gitlab.fritz.box/GevatterGaul/blog.git
23:38:25.367827 run-command.c:664       trace: run_command: git-remote-https origin https://gitlab.fritz.box/GevatterGaul/blog.git
23:38:25.373542 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git-remote-https
23:38:25.374073 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.376194 http.c:756              == Info: Couldn't find host gitlab.fritz.box in the .netrc file; using defaults
23:38:25.383839 http.c:756              == Info:   Trying 10.42.1.21...
23:38:25.383856 http.c:756              == Info: TCP_NODELAY set
23:38:25.419097 http.c:756              == Info: Connected to gitlab.fritz.box (10.42.1.21) port 443 (#0)
23:38:25.419208 http.c:756              == Info: ALPN, offering h2
23:38:25.419217 http.c:756              == Info: ALPN, offering http/1.1
23:38:25.426185 http.c:756              == Info: successfully set certificate verify locations:
23:38:25.426198 http.c:756              == Info:   CAfile: /etc/ssl/cert.pem
  CApath: none
23:38:25.426265 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
23:38:25.431344 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
23:38:25.431371 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
23:38:25.442715 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
23:38:25.442815 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
23:38:25.443033 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
23:38:25.443040 http.c:756              == Info: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
23:38:25.443066 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
23:38:25.446304 http.c:756              == Info: TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
23:38:25.446344 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Finished (20):
23:38:25.446356 http.c:756              == Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
23:38:25.446359 http.c:756              == Info: ALPN, server accepted to use h2
23:38:25.446363 http.c:756              == Info: Server certificate:
23:38:25.446374 http.c:756              == Info:  subject: C=DE; ST=Baden-Wuerttemberg; L=Karlsruhe; O=Behringer; CN=gitlab.fritz.box
23:38:25.446378 http.c:756              == Info:  start date: Apr 11 22:31:27 2021 GMT
23:38:25.446382 http.c:756              == Info:  expire date: Apr 11 22:31:27 2022 GMT
23:38:25.446391 http.c:756              == Info:  subjectAltName: host "gitlab.fritz.box" matched cert's "gitlab.fritz.box"
23:38:25.446399 http.c:756              == Info:  issuer: C=DE; ST=Baden-Wuerttemberg; O=Behringer; CN=Behringer
23:38:25.446402 http.c:756              == Info:  SSL certificate verify ok.
23:38:25.446419 http.c:756              == Info: Using HTTP2, server supports multi-use
23:38:25.446422 http.c:756              == Info: Connection state changed (HTTP/2 confirmed)
23:38:25.446427 http.c:756              == Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
23:38:25.446516 http.c:756              == Info: Using Stream ID: 1 (easy handle 0x7fc060008200)
23:38:25.446539 http.c:703              => Send header, 0000000199 bytes (0x000000c7)
23:38:25.446565 http.c:715              => Send header: GET /GevatterGaul/blog.git/info/refs?service=git-receive-pack HTTP/2
23:38:25.446568 http.c:715              => Send header: Host: gitlab.fritz.box
23:38:25.446570 http.c:715              => Send header: User-Agent: git/2.30.1 (Apple Git-130)
23:38:25.446573 http.c:715              => Send header: Accept: */*
23:38:25.446575 http.c:715              => Send header: Accept-Encoding: deflate, gzip
23:38:25.446577 http.c:715              => Send header: Pragma: no-cache
23:38:25.446579 http.c:715              => Send header:
23:38:25.446596 http.c:756              == Info: Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
23:38:25.473948 http.c:703              <= Recv header, 0000000013 bytes (0x0000000d)
23:38:25.473962 http.c:715              <= Recv header: HTTP/2 401
23:38:25.473966 http.c:703              <= Recv header, 0000000025 bytes (0x00000019)
23:38:25.473969 http.c:715              <= Recv header: cache-control: no-cache
23:38:25.473972 http.c:703              <= Recv header, 0000000041 bytes (0x00000029)
23:38:25.473975 http.c:715              <= Recv header: content-type: text/plain; charset=utf-8
23:38:25.473977 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
23:38:25.473980 http.c:715              <= Recv header: date: Mon, 22 Nov 2021 22:38:25 GMT
23:38:25.473983 http.c:703              <= Recv header, 0000000015 bytes (0x0000000f)
23:38:25.473985 http.c:715              <= Recv header: server: nginx
23:38:25.473988 http.c:703              <= Recv header, 0000000014 bytes (0x0000000e)
23:38:25.473990 http.c:715              <= Recv header: vary: Accept
23:38:25.473993 http.c:703              <= Recv header, 0000000040 bytes (0x00000028)
23:38:25.474061 http.c:715              <= Recv header: www-authenticate: Basic realm="GitLab"
23:38:25.474069 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
23:38:25.474073 http.c:715              <= Recv header: x-content-type-options: nosniff
23:38:25.474099 http.c:703              <= Recv header, 0000000028 bytes (0x0000001c)
23:38:25.474101 http.c:715              <= Recv header: x-download-options: noopen
23:38:25.474104 http.c:703              <= Recv header, 0000000029 bytes (0x0000001d)
23:38:25.474106 http.c:715              <= Recv header: x-frame-options: SAMEORIGIN
23:38:25.474109 http.c:703              <= Recv header, 0000000041 bytes (0x00000029)
23:38:25.474111 http.c:715              <= Recv header: x-permitted-cross-domain-policies: none
23:38:25.474113 http.c:703              <= Recv header, 0000000042 bytes (0x0000002a)
23:38:25.474115 http.c:715              <= Recv header: x-request-id: 01FN4WST7CTXXJJAKEKZNJM8D6
23:38:25.474226 http.c:703              <= Recv header, 0000000021 bytes (0x00000015)
23:38:25.474234 http.c:715              <= Recv header: x-runtime: 0.020858
23:38:25.474237 http.c:703              <= Recv header, 0000000033 bytes (0x00000021)
23:38:25.474240 http.c:715              <= Recv header: x-xss-protection: 1; mode=block
23:38:25.474243 http.c:703              <= Recv header, 0000000020 bytes (0x00000014)
23:38:25.474245 http.c:715              <= Recv header: content-length: 26
23:38:25.474249 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
23:38:25.474253 http.c:715              <= Recv header:
23:38:25.474311 http.c:756              == Info: Connection #0 to host gitlab.fritz.box left intact
23:38:25.474400 run-command.c:664       trace: run_command: 'git credential-osxkeychain get'
23:38:25.482498 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
23:38:25.483117 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.483695 git.c:730               trace: exec: git-credential-osxkeychain get
23:38:25.483742 run-command.c:664       trace: run_command: git-credential-osxkeychain get
23:38:25.509132 http.c:756              == Info: Found bundle for host gitlab.fritz.box: 0x7fc05f904d20 [can multiplex]
23:38:25.509165 http.c:756              == Info: Re-using existing connection! (#0) with host gitlab.fritz.box
23:38:25.509179 http.c:756              == Info: Connected to gitlab.fritz.box (10.42.1.21) port 443 (#0)
23:38:25.509196 http.c:756              == Info: Server auth using Basic with user 'GevatterGaul'
23:38:25.509241 http.c:756              == Info: Using Stream ID: 3 (easy handle 0x7fc060008200)
23:38:25.509343 http.c:703              => Send header, 0000000274 bytes (0x00000112)
23:38:25.509352 http.c:715              => Send header: GET /GevatterGaul/blog.git/info/refs?service=git-receive-pack HTTP/2
23:38:25.509355 http.c:715              => Send header: Host: gitlab.fritz.box
23:38:25.509357 http.c:715              => Send header: Authorization: Basic <redacted>
23:38:25.509362 http.c:715              => Send header: User-Agent: git/2.30.1 (Apple Git-130)
23:38:25.509364 http.c:715              => Send header: Accept: */*
23:38:25.509366 http.c:715              => Send header: Accept-Encoding: deflate, gzip
23:38:25.509369 http.c:715              => Send header: Pragma: no-cache
23:38:25.509370 http.c:715              => Send header:
23:38:25.601638 http.c:703              <= Recv header, 0000000013 bytes (0x0000000d)
23:38:25.601651 http.c:715              <= Recv header: HTTP/2 200
23:38:25.601655 http.c:703              <= Recv header, 0000000025 bytes (0x00000019)
23:38:25.601658 http.c:715              <= Recv header: cache-control: no-cache
23:38:25.601667 http.c:703              <= Recv header, 0000000024 bytes (0x00000018)
23:38:25.601669 http.c:715              <= Recv header: content-encoding: gzip
23:38:25.601672 http.c:703              <= Recv header, 0000000060 bytes (0x0000003c)
23:38:25.601674 http.c:715              <= Recv header: content-type: application/x-git-receive-pack-advertisement
23:38:25.601677 http.c:703              <= Recv header, 0000000037 bytes (0x00000025)
23:38:25.601679 http.c:715              <= Recv header: date: Mon, 22 Nov 2021 22:38:25 GMT
23:38:25.601682 http.c:703              <= Recv header, 0000000050 bytes (0x00000032)
23:38:25.601684 http.c:715              <= Recv header: referrer-policy: strict-origin-when-cross-origin
23:38:25.601696 http.c:703              <= Recv header, 0000000015 bytes (0x0000000f)
23:38:25.601698 http.c:715              <= Recv header: server: nginx
23:38:25.601700 http.c:703              <= Recv header, 0000000045 bytes (0x0000002d)
23:38:25.601702 http.c:715              <= Recv header: strict-transport-security: max-age=63072000
23:38:25.601705 http.c:703              <= Recv header, 0000000021 bytes (0x00000015)
23:38:25.601707 http.c:715              <= Recv header: content-length: 195
23:38:25.601710 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
23:38:25.601712 http.c:715              <= Recv header:
23:38:25.601760 http.c:756              == Info: Connection #0 to host gitlab.fritz.box left intact
23:38:25.601774 run-command.c:664       trace: run_command: 'git credential-osxkeychain store'
23:38:25.606895 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
23:38:25.607684 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.608307 git.c:730               trace: exec: git-credential-osxkeychain store
23:38:25.608337 run-command.c:664       trace: run_command: git-credential-osxkeychain store
23:38:25.650675 run-command.c:664       trace: run_command: git send-pack --stateless-rpc --helper-status --thin --progress https://gitlab.fritz.box/GevatterGaul/blog.git/ --stdin
23:38:25.652824 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
23:38:25.653284 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.653839 git.c:444               trace: built-in: git send-pack --stateless-rpc --helper-status --thin --progress https://gitlab.fritz.box/GevatterGaul/blog.git/ --stdin
23:38:25.654969 run-command.c:664       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
23:38:25.657120 exec-cmd.c:139          trace: resolved executable path from Darwin stack: /Library/Developer/CommandLineTools/usr/libexec/git-core/git
23:38:25.657608 exec-cmd.c:238          trace: resolved executable dir: /Library/Developer/CommandLineTools/usr/libexec/git-core
23:38:25.658172 git.c:444               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 138, done.
Counting objects: 100% (138/138), done.
Delta compression using up to 16 threads
Compressing objects: 100% (118/118), done.
Writing objects: 100% (118/118), 45.47 MiB | 28.20 MiB/s, done.
Total 118 (delta 63), reused 0 (delta 0), pack-reused 0
23:38:27.877742 http.c:756              == Info: Found bundle for host gitlab.fritz.box: 0x7fc05f904d20 [can multiplex]
23:38:27.877816 http.c:756              == Info: Re-using existing connection! (#0) with host gitlab.fritz.box
23:38:27.877830 http.c:756              == Info: Connected to gitlab.fritz.box (10.42.1.21) port 443 (#0)
23:38:27.877843 http.c:756              == Info: Server auth using Basic with user 'GevatterGaul'
23:38:27.877866 http.c:756              == Info: Using Stream ID: 5 (easy handle 0x7fc060008200)
23:38:27.877955 http.c:703              => Send header, 0000000353 bytes (0x00000161)
23:38:27.878009 http.c:715              => Send header: POST /GevatterGaul/blog.git/git-receive-pack HTTP/2
23:38:27.878014 http.c:715              => Send header: Host: gitlab.fritz.box
23:38:27.878036 http.c:715              => Send header: Authorization: Basic <redacted>
23:38:27.878039 http.c:715              => Send header: User-Agent: git/2.30.1 (Apple Git-130)
23:38:27.878041 http.c:715              => Send header: Accept-Encoding: deflate, gzip
23:38:27.878044 http.c:715              => Send header: Content-Type: application/x-git-receive-pack-request
23:38:27.878047 http.c:715              => Send header: Accept: application/x-git-receive-pack-result
23:38:27.878049 http.c:715              => Send header: Content-Length: 47678527
23:38:27.878051 http.c:715              => Send header:
23:38:28.157588 http.c:756              == Info: TLSv1.2 (IN), TLS alert, bad record mac (532):
23:38:28.157626 http.c:756              == Info: LibreSSL SSL_read: error:1404C3FC:SSL routines:ST_OK:sslv3 alert bad record mac, errno 0
23:38:28.157633 http.c:756              == Info: Failed receiving HTTP2 data
23:38:28.157638 http.c:756              == Info: Connection died, retrying a fresh connect
23:38:28.157681 http.c:756              == Info: Closing connection 0
23:38:28.157730 http.c:756              == Info: TLSv1.2 (OUT), TLS alert, close notify (256):
23:38:28.160202 http.c:756              == Info: Issue another request to this URL: 'https://gitlab.fritz.box/GevatterGaul/blog.git/git-receive-pack'
23:38:28.160241 http.c:756              == Info: NTLM-proxy picked AND auth done set, clear picked!
23:38:28.160261 http.c:756              == Info: Hostname gitlab.fritz.box was found in DNS cache
23:38:28.160331 http.c:756              == Info:   Trying 10.42.1.21...
23:38:28.160342 http.c:756              == Info: TCP_NODELAY set
23:38:28.184258 http.c:756              == Info: Connected to gitlab.fritz.box (10.42.1.21) port 443 (#1)
23:38:28.184396 http.c:756              == Info: ALPN, offering h2
23:38:28.184411 http.c:756              == Info: ALPN, offering http/1.1
23:38:28.193966 http.c:756              == Info: successfully set certificate verify locations:
23:38:28.193980 http.c:756              == Info:   CAfile: /etc/ssl/cert.pem
  CApath: none
23:38:28.194115 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Client hello (1):
23:38:28.202177 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server hello (2):
23:38:28.202228 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
23:38:28.204886 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
23:38:28.205025 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
23:38:28.205381 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
23:38:28.205396 http.c:756              == Info: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
23:38:28.205438 http.c:756              == Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
23:38:28.207657 http.c:756              == Info: TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
23:38:28.207737 http.c:756              == Info: TLSv1.2 (IN), TLS handshake, Finished (20):
23:38:28.207756 http.c:756              == Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
23:38:28.207761 http.c:756              == Info: ALPN, server accepted to use h2
23:38:28.207768 http.c:756              == Info: Server certificate:
23:38:28.207783 http.c:756              == Info:  subject: C=DE; ST=Baden-Wuerttemberg; L=Karlsruhe; O=Behringer; CN=gitlab.fritz.box
23:38:28.207790 http.c:756              == Info:  start date: Apr 11 22:31:27 2021 GMT
23:38:28.207795 http.c:756              == Info:  expire date: Apr 11 22:31:27 2022 GMT
23:38:28.207812 http.c:756              == Info:  subjectAltName: host "gitlab.fritz.box" matched cert's "gitlab.fritz.box"
23:38:28.207824 http.c:756              == Info:  issuer: C=DE; ST=Baden-Wuerttemberg; O=Behringer; CN=Behringer
23:38:28.207830 http.c:756              == Info:  SSL certificate verify ok.
23:38:28.207854 http.c:756              == Info: Using HTTP2, server supports multi-use
23:38:28.207859 http.c:756              == Info: Connection state changed (HTTP/2 confirmed)
23:38:28.207865 http.c:756              == Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
23:38:28.207933 http.c:756              == Info: Server auth using Basic with user 'GevatterGaul'
23:38:28.207959 http.c:756              == Info: Using Stream ID: 1 (easy handle 0x7fc060008200)
23:38:28.207989 http.c:703              => Send header, 0000000353 bytes (0x00000161)
23:38:28.207999 http.c:715              => Send header: POST /GevatterGaul/blog.git/git-receive-pack HTTP/2
23:38:28.208004 http.c:715              => Send header: Host: gitlab.fritz.box
23:38:28.208007 http.c:715              => Send header: Authorization: Basic <redacted>
23:38:28.208010 http.c:715              => Send header: User-Agent: git/2.30.1 (Apple Git-130)
23:38:28.208013 http.c:715              => Send header: Accept-Encoding: deflate, gzip
23:38:28.208017 http.c:715              => Send header: Content-Type: application/x-git-receive-pack-request
23:38:28.208020 http.c:715              => Send header: Accept: application/x-git-receive-pack-result
23:38:28.208023 http.c:715              => Send header: Content-Length: 47678527
23:38:28.208026 http.c:715              => Send header:
23:38:28.209222 http.c:756              == Info: Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
23:38:28.407818 http.c:756              == Info: TLSv1.2 (IN), TLS alert, bad record mac (532):
23:38:28.407852 http.c:756              == Info: LibreSSL SSL_read: error:1404C3FC:SSL routines:ST_OK:sslv3 alert bad record mac, errno 0
23:38:28.407858 http.c:756              == Info: Failed receiving HTTP2 data
23:38:28.407880 http.c:756              == Info: SSL_write() returned SYSCALL, errno = 32
23:38:28.407885 http.c:756              == Info: Failed sending HTTP2 data
23:38:28.407896 http.c:756              == Info: Connection #1 to host gitlab.fritz.box left intact
error: RPC failed; curl 56 LibreSSL SSL_read: error:1404C3FC:SSL routines:ST_OK:sslv3 alert bad record mac, errno 0
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

Read More

Qwen3 and Gemma3 Performance on Consumer Hardware

2025-06-26

Large Language Models (LLMs), or AI, are widely adopted by now. Most of us use it daily for work, leisure and private matters. That also means that we share a lot of possibly sensitive information with the LLM. This can be problematic when using 3rd party services like ChatGPT, especially now that OpenAI has to retain all chat logs indefinitely. So how about running an LLM on your own hardware?

We previously measured two popular current-generation llama 3 and deepseek-r1. This time we follow up with Qwen3 and Gemma3, both released in the last months. We want to give you an idea which kind of model will run on what hardware and what kind of performance you can expect.

Test Setup

We test two of the currently popular LLM families, Qwen3 and Gemma3, in quantized and 16bit floating point (fp16) versions for their performance on consumer hardware.
We run the models using ollama 0.9.2 and open-webui as frontend. We collect the measured response tokens after executing the following query:

I need a summary of the book “War and Peace¨. Please write at least 500 words.

For each test, the model fits into the memory of the graphics card (VRAM). Mobile devices are allowed to cool down before each test run.

For Qwen3, the thinking mode was left at default, which is “on”.

Tested Models

In this article we focus on two of the currently popular LLMs:

  • Qwen3, developed by Alibaba, is optimized for deployment on Alibaba Cloud infrastructure, leveraging GPUs and specialized AI accelerators. It excels in advanced conversational AI, content generation, multilingual interactions, and complex reasoning tasks. Typical applications include customer support automation, virtual assistants, translation, summarization, and sentiment analysis. Its scalable architecture ensures efficient integration into cloud-based business solutions, particularly within Alibaba’s ecosystem.

  • Gemma3, created by Google DeepMind, is designed for versatility and efficiency across diverse hardware platforms, including consumer-grade GPUs, CPUs, and edge devices. It effectively handles tasks such as text generation, summarization, conversational AI, and question-answering. Its lightweight, open-source architecture makes it ideal for resource-constrained environments, enabling applications like personal assistants, educational tools, and interactive chatbots. Gemma3’s open-source nature encourages customization, experimentation, and broad adoption in both research and industry contexts.

Model Variant Precision VRAM Size
qwen3:0.6b Q4_K_M 2.2GB
qwen3:4b Q4_K_M 5.2GB
qwen3:8b Q4_K_M 7.5GB
qwen3:14b Q4_K_M 12GB
qwen3:32b Q4_K_M 25GB
qwen3:0.6b fp16 3GB
qwen3:1.7b fp16 5.3GB
qwen3:4b fp16 10GB
qwen3:8b fp16 18GB
qwen3:14b fp16 32GB
gemma3:1b instruct Q4_K_M 1.9GB
gemma3:4b instruct Q4_K_M 6GB
gemma3:12b instruct Q4_K_M 11GB
gemma3:27b instruct Q4_K_M 21GB
gemma3:1b instruct, quantization aware Q4_K_M 2.1GB
gemma3:4b instruct, quantization aware Q4_K_M 6.6GB
gemma3:12b instruct, quantization aware Q4_K_M 12GB
gemma3:27b instruct, quantization aware Q4_K_M 22GB
gemma3:1b instruct fp16 3.1GB
gemma3:4b instruct fp16 11GB
gemma3:12b instruct fp16 31GB
gemma3:27b instruct fp16 63GB
Table 1: tested llms

Table 1 shows the tested LLMs. Ollama uses quantized models by default. If you specify a more detailed tag you can choose different quantizations as well as the non-quantized floating point versions. We select Qwen3 and Gemma3 for their popularity and start with the smallest versions parameter-wise. We continuously choose higher parameter models until they would not fit into any of the test systems VRAM.
The last column in the table shows the size of the model when loaded into VRAM as reported by ollama ps. We can already see here that not all models will fit into the VRAM of the test systems from Table 2.

Read More

Typescript x AI Meetup Talk: Privacy First AI Summarizing in TypeScript

2025-06-04

On May 21, 2025 I was fortunate enough to present a TNG hacking project at the Munich Typescript Meetup. Thanks to the organizer Carl Assmann and TNG Technology Consulting GmbH for having me!

The Typescript x AI organizers and speakers: Luca Becker, Carl Assmann, Luisa Peter, Lucas L. Treffenstädt, Benjamin Behringer, Alexander Opalic, Johannes Loher
The Typescript x AI organizers and speakers: Luca Becker, Carl Assmann, Luisa Peter, Lucas L. Treffenstädt, Benjamin Behringer, Alexander Opalic, Johannes Loher

Talk Abstract

Nearly every AI-enabled product in 2025 has a summarizing function. But they all use 3rd party software to get the summary. This raises major privacy concerns and is prohibitive for most of us in the community. We display a small React/Typescript app that will enable you to summarize even longer text on your local machine or anywhere and a Large Language Model of your choosing. We motivate the app and show it in a live demo with local and remote LLMs. We discuss different approaches to prompting and the quality of summaries on various LLMs. We show the implementation and dive into the most interesting parts of the code.

Future Work

We are currently developing the app and plan to release a public version in the future. We will keep you posted on the TNG Twitter account, the Bored Consultant Twitter account or my LinkedIn Page.

Read More

LLM Performance on Consumer Hardware

2025-05-03

LLMs are huge, slow to run and require terrabytes of RAM on the newest graphics cards, right? Well, maybe not. In this article we compare the performance of two popular current LLMs: llama 3.x and deepseek-r1 on a variety of consumer hardware from Laptop GPUs to dual Nvidia 5090s. We test quantized and full precision models and show which one can fit into the memory of your graphics card. We also test against Apples M chip series and explore what can be achieved with older but cheaper server hardware.

Test Setup

We test two of the currently popular LLM families, llama 3 and deepseek-r1, in quantized and 16bit floating point (fp16) versions for their performance on consumer hardware.
We run the models using ollama 0.6.5 and open-webui as frontend. We collect the measured response tokens after executing the following query:

I need a summary of the book “War and Peace¨. Please write at least 500 words.

For each test, the model fits into the memory of the graphics card (VRAM). Mobile devices are allowed to cool down before each test run.

Tested Models

In this article we focus on two of the currently popular LLMs:

  • Llama 3, developed by Meta, represents the latest evolution in open-source large language models, building upon its predecessors with enhanced capabilities, improved efficiency, and greater contextual understanding. It comes in various sizes, typically ranging from smaller, more efficient models suitable for edge computing and personal devices, to larger, more powerful variants designed for complex tasks and extensive data processing. Its versatility makes it ideal for applications such as chatbots, content generation, coding assistance, and research. For the AI community, Llama 3 signifies a significant step toward democratizing advanced AI technology, fostering innovation, and enabling broader access to sophisticated AI tools and research opportunities.
  • DeepSeek-R1, developed by DeepSeek AI, is a cutting-edge large language model designed specifically to excel in coding and technical tasks. It comes in multiple variants, including models optimized for general-purpose programming, debugging, and software development assistance, making it highly versatile for developers and tech professionals. Its primary uses include code generation, error detection, automated debugging, and providing detailed technical explanations. For the AI community, DeepSeek-R1 represents a significant advancement in specialized AI models, enhancing productivity and accuracy in software development, and contributing to the broader adoption of AI-driven coding solutions.
Model Variant Precision VRAM Size
llama3.2:1b instruct Q8_0 2.7GB
llama3.2:3b instruct Q4_K_M 4GB
llama3.1:8b instruct Q4_K_M 6.9GB
llama3.3:70b instruct Q4_K_M 49GB
llama3.2:1b instruct fp16 3.9GB
llama3.2:3b instruct fp16 8.5GB
llama3.1:8b instruct fp16 17GB
deepseek-r1:7b qwen distill Q4_K_M 6GB
deepseek-r1:8b llama distill Q4_K_M 6.9GB
deepseek-r1:14b qwen distill Q4_K_M 11GB
deepseek-r1:32b qwen distill Q4_K_M 25GB
deepseek-r1:70b qwen distill Q4_K_M 49GB
deepseek-r1:1.5b qwen distill fp16 4.2GB
deepseek-r1:7b qwen distill fp16 16GB
deepseek-r1:14b qwen distill fp16 32GB
Table 1: tested llms

Table 1 shows the tested LLMs. Ollama uses quantized models by default. If you specify a more detailed tag you can choose different quantizations as well as the non-quantized floating point versions. We select llama 3.x and deepseek-r1 for their popularity and start with the smallest versions parameter-wise. We continuously choose higher parameter models until they would not fit into any of the test systems VRAM.
The last column in the table shows the size of the model when loaded into VRAM as reported by ollama ps. We can already see here that not all models will fit into the VRAM of the test systems from Table 2.

Read More

The Rebuttal: AI's Impact On Content Creation And Knowledge Sharing

2025-04-22

In our earlier post “Will AI Kill The Blogosphere?” we discussed the impact of AI on content creation in the internet and how to react to this change. Now we want to find out what the AI itself thinks of the subject and our previous article. We created a rebuttal using the following prompt:

Read the given article and write a response. Evaluate the premise and conclusion
in detail. Draw your own conclusion on the subject. Be honest and don´t hold
back. Try to provide value for the reader.

The following the Grok 3’s answer. Enjoy!

Human and AI arguing

Response to the Article: AI’s Impact on Content Creation and Knowledge Sharing

The article “The Impact of AI on Content Creation and Knowledge Sharing” explores how artificial intelligence is reshaping the landscape of content creation and the dissemination of knowledge. Below, I’ll evaluate the article’s premise and conclusion in detail, offer my own perspective, and provide actionable insights for readers navigating this evolving terrain. Let’s dive in.

Read More