Published in Technology on A Place For My Head. Tagged , , , , , & . #29 in the Colophon: Finding A Place For My Head series.

I wanted to take control of my images instead of continuing to use Netlify Large Media (built on Git LFS), so I read the Netlify support guide. It didn’t explain what local backups meant here:

Please be absolutely certain you have local backups of all required files before directing our support team to uninstall the Large Media add-on from your site.

I assumed my local clone, which I’d been using since I first enabled Large Media, would suffice. I rushed into requesting the removal. (The thread was deleted, but I archived it.) A few minutes later, I added questions about the mechanism and an explanation of what I understood local backups were. I ran git lfs migrate export, thinking it would safeguard my data.

I woke up the next day to find the request had been processed and the Support Engineer had ignored my questions. I typed in the commands they suggested running after the migration, revealing that I had lost historical data (images in old commits). I found out I was actually supposed to run git lfs pull --everything origin before the migration. I now only had the local versions of some images. I also remembered with a sinking feeling that I’d lost offsite historical backups (via a third party) a year or so ago when the RBI’s recurring payments fiasco led my subscription to lapse.

Still, it was only historical data, and I’d lost it through my own fault. I decided to repair the repository manually. I poked around a bit but couldn’t figure out how it worked, so I asked in the thread about inserting files manually. That got me a generic reply about tracking and untracking files. Realizing I wouldn’t get any help here, I politely but clearly pointed out they weren’t reading what I wrote:

I’m sorry, but it doesn’t look to me like you’ve read more than a few words here and there of either of my messages, given that you ignored my questions in the first one (answering which would have prevented me from losing data because of my misunderstanding) and are now answering a different question from the one I asked. I’ll try to figure it out on my own. Thanks anyway.

I thought that would be the end of it.

A puzzlingly defensive reaction

Within 24 hours, I was notified of a reply:

Slow your roll there. There is no need to get rude and I’m also fairly certain you should still have all the Git LFS data.

I do agree I did miss one (and only one) question from you but that doesn’t justify being insulting.

You said you were ready to proceed with the Large Media add-on removal. You did not say to wait until you were sure you were ready. You wrote this:

Please remove the Large Media extension from my site at https://shivjm.blog.

You said to remove it and we did so. If you lost data, that is not my fault. We told you to be sure before making the post.

The answer to your question (the only question and one that I missed) is that you should have run this command:

git lfs fetch --all

Naturally, the correct command was now at their fingertips. The remainder of the message was useless advice. It closed with:

I hope this helps. If not, please feel free to reply anytime as long as you are respectful when doing so.

I was shocked then annoyed by the condescending tone and complete refusal to either read what I wrote or acknowledge their own mistakes. I replied at length:

Let’s be clear about a few things, please:

  1. Telling you (correctly, I hope we can both agree after this) that you didn’t read my messages thoroughly is neither rude nor disrespectful. As a fellow IT professional with past experience in support, I was extremely careful to couch it in bland terms while still conveying my disappointment. You might not like it being pointed out, but that is not the same thing as my insulting you. I’m quite perplexed to hear this.

  2. As regards ‘one (and only one question)’, I can see that ‘I’m not sure whether that qualifies as having local backups or not’ should have been phrased more explicitly as a question, and perhaps it would be more accurate for me to say ‘concerns’ rather than ‘questions’.

  3. If you lost data, that is not my fault. We told you to be sure before making the post.

    This is a response to something I didn’t say. I made the mistake and the responsibility is mine; my second message asking about the store is entirely about recovering. Meanwhile, here’s what I did say in my last message:

    you ignored my questions in the first one (answering which would have prevented me from losing data because of my misunderstanding)

    Had you clarified that a clone of the repository is insufficient, I would have asked you to wait. The fact that you didn’t doesn’t make you responsible, since I’m still the one who asked you to do it. It just means you had an opportunity to stop me from making a mistake.

I must say, I was a big fan of Netlify before this. After my botched migration, I would simply have suggested clarifying the bit about local backups in the docs… had we ever reached that point. Even your second reply only resigned me to continuing my attempts to repair the history on my own, which I honestly wouldn’t expect Netlify to do much about anyway, given that it’s an odd and complex situation arising from my mistakes. However, the defensive response and personally offended tone in this most recent message are hard to understand.

At any rate, coming back to the subject:

If so, your local system should have all copies (past and present) of the Git LFS files as your local repo is the one creating the commits that get pushed. You cannot push a commit without the data existing locally. This means all versions of the Git LFS file must have existed locally at some point.

[snip]

So, in most case even if you didn’t run git lfs fetch --all, the data should exist locally.

[snip]

However, the current versions should still exist.

Yes, the current files do exist. It’s historical files from older commits that are missing. I have no idea why Git LFS deleted the objects locally, as I’m using the same clone as before and have never interacted with LFS beyond adding files. I’ve spent the past few days trying various approaches to edit the old commits to insert placeholder images without triggering a Git LFS fetch and without breaking my history. I’ll keep trying.

Shortly afterwards, a friend told me the engineer had Unlisted the thread, presumably hiding it from search engines and ordinary users of the forums. I didn’t hear from Netlify again.

The fallout

Deleting the Large Media extension made the images go missing on my site entirely. It took a little over a week for me to repair the history enough that I could run netlify deploy from a local build, leaving my pages with full-size images loading in 400-pixel boxes. It was another day before I got eleventy-img working, and another day before I deployed a newer local build to fix the situation. And then it was two more weeks before I could get my build working in GitLab.

Now, to be clear: Netlify Support did as I asked. The mistake was mine. What frustrates and annoys me is the repeated refusal to read what I wrote and the attempt to deflect from their mistakes by pretending I’d said something offensive, when all I did was point out what they were doing.

At any rate, for the record, here are all the reasons why I’m an idiot:

I’ll count myself lucky to have learnt my lesson by losing only a bit of historical data that no one else would miss.

Next in series: (#30 in Colophon: Finding A Place For My Head)