Coding with AI moves so fast that it’s easy to feel left-out, so it was a great opportunity to exchange tips, techniques or new use cases.

I learned a ton during that meetup! Here are a few interesting tidbits:

• Someone connected OpenClaw on a Mac mini and uses it to try-out app ideas. For iOS animations, he asks the agent to build 5 variants of the app. He then downloads each one on his phone to pick the best option

• Someone is building CRON jobs for agents: e.g. when a Sentry alert pops up, the agent can investigate

• Someone explained that they heavily use logs throughout their production Kubernetes cluster and have agents query them. No Datadog or Grafana as it would be expensive with that volume. When an issue arises, all it takes is to ask the agent to go investigate

• The same person explained that they have one big machine where developers run their sessions on tmux (vs running in the Cloud)

People also shared a bunch of great tools I never heard about:

• Workmux: a tool for managing git worktrees and tmux windows as isolated development environments

• Git Butler: an agent orchestrator that makes running agents on multiple branches in parallel

Given the pace of AI, I’m thinking of running this event about once a month. If you have a great setup you’d like to showcase, please send me a message! Since every attendee presents, spots are limited.

1. The agents were not sandboxed, they were all running on the same Mac VM

2. The setup was not frictionless, requiring some fiddling to open a terminal on a machine or connect with VSCode

3. I had to secure the Cloudflare tunnel myself, OpenCode Portal doesn’t provide any auth

After reading my article, a friend told me that Coder could solve a lot of my problems. After struggling to look it up because of their impossible-to-google name, I spent the weekend trying their new Tasks feature. In a nutshell, you install Coder on a VM, it spins up a web UI where you can launch background tasks. Each task is launched inside its own Docker container (but you can also use Kubernetes).

At your fingertips, the web UI shows the holy trinity of the vibe coder: Claude Code, Cursor, a terminal.

Coder has other nifty features. You can configure templates that control your agent environments (e.g dependencies, access tokens), presets for templates parameters (I have a preset with a branch I use often instead of main), the GitHub integration is quite smooth (even for GitHub Organizations), it can tell you why an environment is slow to start:

So what’s missing? The web UI is not usable on mobile for anything besides checking the status. But the beauty of open-source is that you can go and fix it with a pull request. You could also build your own UI since they have an API (if you do build a great one for iPhone, please ping me!). Also, they have a few interesting features, such as pre-warming the environments, that require talking to their sales team.

We’re almost there! If you compare to Ramp’s setup, we’re missing a few items: instant startup (by the time you finish typing), spawning sub-agents (so an agent can try 10 ideas in parallel), a great mobile UI and a Slack interface.

As per Gas Town’s creator, I’m about to reach stage 7 of AI-Assisted coding: 10+ agents, hand-managed. Future will tell whether I reach stage 8, building my own agent orchestrator. Where does forking an existing orchestrator put you?

The app is: free, fast, without any tracking or analytics (I checked), has notifications for posts, Algolia search and is extremely customizable.

You can select how minimalist you want the home page to be, what gestures are on, how the web pages should open. How I configured my home page as well as a sample of the numerous settings:

How do I know it doesn’t collect any data? I checked with Proxyman. I can’t guarantee it there’s no secret exfiltration method, but I found the app setup interesting. Since there’s no official Hacker News API, the app has to work with Hacker News’ HTML directly: it fetches all the selectors required to parse the content at startup. No server required. Here’s an example:

Get Hack on iOS (also available on Android, although I didn’t try it). I have no affiliation with the developer, besides the tips I sent. Dear developer, if you read this blog post, thank you!

• run on remote VMs (so you can close your laptop)

• spawn on-demand before you finish typing your prompt (so you can try any idea quickly)

• have access to all of Ramp’s tools (Sentry, Datadog)

• can spawn sub-agents (in a new VM) and are multiplayer (other humans can view the agent’s work)

• are multi-interface: Slack, web UI, voice

If you follow Steve Yegge’s (Gas Town’s author) classification, Ramp’s developers are at the final level of agent building:

Over the weekend, I attempted to rebuild Ramp’s system by wiring open-source tools. For this first version, I reduced the scope: no multiplayer, can’t spawn new VMs. But I added a twist: I wanted to test features in our iOS app, therefore I needed to run macOS.

Last night, I asked Claude to benchmark our iOS app’s build time. It researched optimizations, measured the impact of each option and gave me a shiny report. By the time I woke up I ended up with a magic flag that resulted in a +10% speedup.

The setup leverages Tart to manage the macOS VM, Cloudflare Tunnels to connect to services running inside the VM with no hassle, OpenCode for the agent and the beautiful OpenCode Portal to interact with it.

I also use the native Mac Screen Sharing app to connect to the UI. I’m not planning on relying on it much, but it’s very handy to debug issues.

A few gotchas: make sure you run your Mac VM on the instance’s local SSD (but be careful it’s ephemeral), AWS has a 24-hour minimum for Mac Instances (hence a ~$30 minimum charge), Cloudflare Tunnels will expose your agent web UI to the public internet if you’re not careful.

So what’s missing? To reach Ramp’s level we’re missing a few key elements: multiplayer, per-user accounts / isolation and spawning new VMs instantly. That last one might be tricky since Apple enforces the 24h billing rule.

This setup may seem a tad complex, but boy does it feel good to wake up after my agents waited for Xcode to build, not me.

That’s the situation we faced about 2 years ago at Photoroom. We provide editing tools for e-commerce images and we wanted guarantee our users that their content was always in sync, even when their teammates where working on it at the same time.

The good news is that we were already sharing our rendering code between platforms (read about it in our 2021 article). The bad news is that it was written in C. Not the best language to decode payloads back and forth over the network.

We ended up picking Rust and built a cross-platform live-collaboration engine. It’s been live (no pun intended) on all 3 platforms for a little while now. Here’s a quick demo:

Why Rust? Why not React Native?

Besides attempting to make it to the home page of Hacker News, we picked Rust because… it would be easier to hire than for C++. We know how attractive the language can be to experienced developers. We also considered Golang, but its runtime makes it more complicated to run on the web.

This may seem like a lot of pointless work to run on 3 platforms when solutions like React Native exist. We strongly believe in providing the very best image editing experience to our users, and we don’t think it’s possible to do so with React Native (but this stance might change in the future).

How it works

In a nutshell, the Rust code is compiled on iOS, Android and the web. The rendering of the UI is done natively (SwiftUI, Compose, React), same for the network calls (URLSession, OkHttp, fetch), same fore rendering (Metal, OpenGL, WebGL/WebGPU through wgpu). But all the logic is inside the Rust code.

Wait, that’s a bit lightweight for an explanation, isn’t it? We’re quite proud of the work we did, that’s why we detailed the nitty-gritty in a 5-parts blog post series.

If this sort of work sounds exciting to you, we’re hiring a head of cross-platform to provide a seamless experience to millions of users. Learn more here.

At Photoroom, we used to increment the major version number on big releases to celebrate. Recently, however, we moved our mobile apps to a versioning scheme like like 2025.51.01 (so [year].[weekNumber].[iteration])

The main reason is convenience. At a glance you can know whether a user contacting support is on an old version or not or if a coworker reporting a bug forgot to update.

It’s also quite useful to understand version distribution. You can see below that most users on iOS are on version 2025.50.03 (meaning the third iteration of the 50th week of the year) and are progressively upgrading to 2025.51.01:

One convenient thing you can do is deprecate some routes by looking at the version number to know which clients are older than one year/some duration, here’s some pseudocode:

version = request.headers["pr-app-version"]
weeksSinceAppWasReleased = parseWeeksSinceAppRelease(version)
if weeksSinceAppWasReleased >= 52:
   return {error: "APP_OUTDATED", errorMessage: "Your app is outdated, please visit the app store to update"}

The downsides

As App Store version have to be incremental, so once you try this setup, you can’t go back.

When we implemented it, I also worried that such long version numbers would be a bit scary for users on the app store listing page, but it seems they don’t mind. On my side, I’m still a bit worried when I see Uber’s super long version number:

Overall, I’d recommend this setup in a heartbeat if you run a mobile app at scale.

This article was cross-posted on Photoroom’s blog

We usually run A/B tests comparing: export rate, time to export, paywall conversion, some forms of user ratings, export retention.

This became such a habit that we thought it would be useful to share our setup.

Our A/B testing setup

On every single request we receive, we fetch the LLM models from our A/B test provider (today we use Amplitude Experiments):

A payload will usually look like this:

{llm: "google/gemini-2.5-flash", fallbackLLM: "openai/gpt-4.1-nano"}

The benefits of this system are that product managers are independent when picking the models for each feature (assuming the provider has been implemented first). Since the flag is fetched for every user on every request, updates are instantaneous.

Why a fallback? The LLM operations are so key to our app that we can’t be down every time OpenAI/Gemini is down. Therefore, every time we specify a model, we also pick its counterpart at another provider. In case you haven’t noticed, LLM providers’ status page tend to look like a Christmas tree light strip (underlying how hard it is to run inference at scale):

Another reason is that even when the API is up, the error rate is between 0.1% and 1%. So you definitely need a fallback (or some form of retries).

But wait! Won’t the fallback impact the results? Even when factoring-in outages, the number of requests falling back is below 1-2%, so we usually don’t take it into account in the A/B test (but we could, by adding the model in the generation event)

What about added latency? We clocked it at a 20ms median. This is because we call our A/B test provider on every request.

Conclusion

You’re most likely using the wrong LLM for your use case. Given how fast the models are improving, you can probably pick a newer generation and it’ll be either faster or cheaper with the same performance.

The only way to make this happen at your company is to make it stupidly easy to try out a new model. For that you need 1. instant changes (no deploy needed), 2. reliability and 3. a clear outcome to measure.

This blog post was cross-posted on Photoroom’s blog

This cable is my “peace of mind” go-to: I throw a single one in my bag instead of four different ones. People at the office call it “the magic cable”: it can transfer data and power from any device to any device:

I bought (and lost) a lot of those cables over the years, with no preference on the vendor. Just look for “multi-usb cable” online

Make sure you don’t buy the one below however. While they can sometimes charge multiple devices at once, these won’t be compatible with data transfer (needed for CarPlay, mobile hotspot, app development).

This article was written at the end of 2024. We have since then migrated our infrastructure to Kubernetes

We’ve been using Django Rest Framework (DRF) for 5 years now at Photoroom. We leverage it to sync users’ projects across clients (iOS, Android, Web), allow team members to collaborate (comments, reactions) and manage billing.

For the first few years, we scaled this backend on a single VM using a few simple tricks. In this article, we’ll share our learnings scaling it to 10+ million monthly active users and ~500 queries per second.

General architecture

As shown in the image above, the setup is dead-simple and quite standard: a reverse proxy (Traefik) in front of DRF. Like all of our servers, the VM is behind Cloudflare’s network to protect it against DDOS and monitor against attacks (more on that later).

The VM boasts a whopping 60 cores, and it started to be quite busy as you can see:

For background tasks, we leverage Celery. We have 3 levels of priorities for our workers:

• Blocking (P1): for tasks that require an response before the request completes, with a latency target of ~100ms. Example: uploading a few assets in parallel before responding

• High priority: for tasks that need to complete in a timely manner but can run late from time to time. Examples: sending notifications after an action is completed or sending events for analytics

• Background: for long running background tasks, some of which can run for a long time (up to a few minutes). For example: sending aggregated analytics, periodic cleanups.

Performance & monitoring

If you’ve played with Django, you probably already know to leverage select_related() / prefetch_related() and carefully add indexes before deploying. But all the carefulness in the world won’t protect you against unexpectedly slow queries. Responses might be fast when the first requests start hitting a route, then get much slower as the table grows.

To prevent those issues, we always carefully monitor for any slowness using an APM (Application Performance Monitoring, for instance Datadog or Dynatrace). Each request is traced and we can analyze the slow paths. Slowness stems from too many many factors: from slow database queries, unoptimized libraries, external services that misbehave.

We have alerts on latency, error rate and throughput. The first 2 are thresholds while the latter leverages anomaly detection. We also have a strict no-error-policy on the Stripe webhooks and get alerted if there’s a single error, regardless of the reason.

The practice of always analyzing and monitoring results in a rather decent performance, with a large majority of the requests below 100ms:

Authentication

In the early days, we picked Firebase for authentication. It saved us countless hours of implementation and maintenance for login mechanism on our 3 platforms (iOS, Android, web). And most likely avoided a few security vulnerabilities. But we ran into a limitation.

As we reached the 100M downloads mark, we ran into a surprise: you can’t have more than 100M anonymous users in Firebase. We leverage anonymous auth to allow our users to explore the app without logging-in, while maintaining a stable user identifier across all of our services. At some point we also made the mistake of adding Firebase auth on our website, adding a few millions visitors to the tally.

We now have alerts and deletion scripts to remove anonymous users who haven’t been active for the past 3 months. Knowing who has been active required a few changes: we didn’t have any reason to store data from non-registered users in our backend. We now have a custom routes that monitors active users.

As it’s relying on Google’s services, our app is also mostly unusable in mainland China. I’m also a bit worried about adding OIDC and SAML: they are part of “Identify Platform”, which would cost us as hefty $70k / month (list price)

Despite those limitations, we don’t regret picking Firebase as it has proven reliable and a huge time-saver.

Leveraging Cloudflare’s cache

We serve around 600TB of data per month, mostly images. This backend proxies them, but we’re in the process of splitting this use case into an independent FastAPI service to ease up on the load. Most of the image requests don’t reach Django, as they are cached by Cloudflare. If we were to use Vercel for the same volume, it would cost us $150k/month just for the traffic [1].

We also cache the content of the homepage of the app, so it is close to the user and the first load is quicker. This sometimes had unexpected consequences: someone updates the homepage from our backoffice, accidentally adds botched content/json, goes back home. A few hours later the cache refreshes and users all around the world start reporting crashes on startup 😢.

An astute reader might raise that an app should never crash decoding JSON. I would oppose that we also use that JSON for rendering images and fonts dynamically, which tends to be more error-prone. Regardless, we’ve since taken actions to much strongly validate the content. We also ensure that Photoroom employees get served un-cached content to be able to spot any issue faster. We also have a Slack command to bust the Cloudflare cache in case it got poisoned with bad content.

Scaling Postgresql

Most people on Hacker News will tell you that Postgres is the only database you’ll need. It’s been true for us. Scaling has mostly been a breeze and it has been mostly forgiving. We still had implement a few tricks.

Trick 1: this one is obvious, but we use a managed DB. Duplicating the DB for test is a breeze. Figma did this, scaling to the maximum of RDS until they were large enough to hire DB experts[3]. Life is too short to manage your database

Trick 2: We disabled long running queries. There is generally no reason for a mobile app to make the user wait more than 15 seconds (user would most likely be gone by then). If not monitored, long running queries can block migrations and other queries. Therefore we prevent queries with the flag POSTGRESQL_STATEMENT_TIMEOUT. Don’t forget to disable the flag for migrations however.

Trick 3: We disabled counts on pagination. Indeed, running a count in Postgres, especially on fast-changing data, can be quite costly. Most very large apps do so, for instance Gmail. We simply query pageSize + 1 items to let the client know if there’s a next page [2]

Trick 4: We acknowledge that we can’t hire people who are both backend experts and database tuners, so we work with an agency with experts on the topic. They tend to be expensive and the level of competency of the person assigned to you varies, but it’s overall useful

Trick 5: Periodic backups with point in time recovery. In case someone messes up badly, we can revert to a few minutes before the incident. Knowing this makes me sleep better at night.

Our dream setup would be to be able to replicate the database with anonymized data + replicate a realistic load pattern, but we have yet to find a simple way to do this

Collaborating with developers on 3 platforms

Photoroom is available on iOS, Android and the web. While we’re working hard on moving most of the REST communication in our Rust multi-platform engine, for a little while each platform still has to write code to communicate with the Django backend.

In the early day, we simply put specs on Notion. Unsurprisingly, this didn’t work very well. We now expose OpenAPI specs, documenting the routes, parameters and responses. We leverage drf-spectacular for this. We also stick to the REST patterns as much as possible, which makes the inner workings of the API more predictable.

We can also monitor the traffic the apps send through custom request headers. For complex serialized json, we also add the name of the platform who serialized it in case some botched JSON goes through the schema validation (botched can also mean non-existing assets, wrong ordering, inconsistent positioning, things that are harder to validate)

Headers to know the platform and who serialized a project. pr-app-version and pr-platform

Mistakes we made

• One day we DDOSed ourselves when a developer misconfigured a deployment and it kept calling a route every 10ms

• We periodically run cleanups for inactive free users’ data. The homepage content of the app is owned by user 0 in the DB. The first time we ran the cleanup, we accidentally deleted the official content since user 0 hadn’t been “active” for a while.

• On another Django project, we accidentally deployed it in another region than the database it was talking to. If you’re looking for ways to make all your REST requests 200ms slower, I recommend this technique

Good decisions we took

• Adding an APM from the very first day, allowing us to debug and investigate slow queries. Today, we couldn’t leave without it

• Adding a linter to prevent non-concurrent indexes adding. Adding index without the CONCURRENTLY keyword locks the table, leading to unnecessary downtime.

• Dedicating time to continuously cleaning and improving the codebase + writing a post mortem on every downtime

Conclusion and what’s next

Overall, Django Rest Framework has been reliable and very forgiving. As Photoroom is crossing the 200 million mobile downloads mark, most our challenges are ahead. Among others, we’ll need to:

• Update the deployment procedure, using a single VM won’t scale much more

• Partition or rework the way we store projects, with the largest Postgresql table being close to 1.5TB

• Update our architecture to support real-time collaboration on top of Django

If this challenge is of interest to you, we’re looking for a Django backend engineer (remote in Europe). Your scope will include this backend as well as the backends that power GenAI features in Photoroom. Email eliot@photoroom.com with your resume, along with a Django trick you learned over the year. You must have extensive Django experience to apply.

[1] Vercel charges us an hefty $250 / TB for our traffic country mix. 250 * 600TB = $150k. Those are list prices, you might be able to get discounts.

[2] You can view more tricks like this one in Squeezing Django performance for 14.9 million users on WhatsApp by Rudi Giesler

[3] https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/

So when someone informed me that the E Ink Corporation was selling 42-inch E Ink panels, I was instantly nerd-sniped. That was the perfect project to fuel my non-traditional display addiction. After the flip-dot display salvaged from a UK bus, the quest for an even bigger build was on.

The result

The frame changing from one image to another. The GIF above doesn’t help with quality, HD video available here

The process

Sourcing components

Some advanced Google-fu was used to find a supplier shipping to Europe. Minimum order quantity: 3 pieces. After a bit of back and forth, the components arrived.

The supplier provided the displays as well as the necessary driving board.

Framing

While the pink packaging certainly has its own charm, we decided to go for something more minimalist. Finding a frame shop that could handle this “special” project was not easy. Some straight-out didn’t answer but one of them was interested in that unconventional build.

So we (well, mostly he) set on to custom build a framing system that could enclose the panel without breaking it and without using glue to make repairs / adjustments easy.

The first build we created didn’t turn out great: the display was slipping between the layers. In the second version, we found a way to keep it sturdy.

Controlling the display

The display controller can be talked to using USB, so I added a Raspberry Pi on the back. It exposes a small webserver that can receive images to display.Back of the frame. Notice how the ribbon cables don’t align: I think I picked the wrong controller board variant

Preprocessing images

The display can only display 16 levels of gray. Displaying images without preprocessing leads to a blocky pattern on gradients in images. Thankfully a few techniques exist. One of them, closer to dark magic than science to me, is called Blue Noise dithering. It consist of adding a blue noise pattern to an image and then thresholding. Simple yet impressive, see for yourself.

Displaying images

If you have ever used a kindle, you’ll know that every few pages it refreshes itself to avoid ghosting. We had the same problem where the previous image would appear slighly faded out, mixed with the new image. The trick is to run a full black then full white image. We found that running 2 full white images did a good enough job, without disrupting the experience too much (see video)

Next steps

The first version of this project now sits in the office. There are a few ideas we’d like to try:

• Make it battery-powered: right now it still requires a power cord. It would allow putting it on a wall without cables

• Ask ChatGPT to generate prompts for us, turning it into our very own art curator

Acknowledgements

A huge thank you to Charly for the idea and support in this building process, to Nico for the framing and to Florian for the dithering code.

This blog post has been discussed on Hacker News