Converting Your Twitter Archive to Markdown

The bird is not well. So it is time to request and download an archive of your Twitter data now, if you haven’t done this recently. After you have requested your archive, it can take a while until you receive it. I requested my archive on the day Elon walked in and got an email notice that my archive is ready for download after about a day. I suppose it might take a little longer at the moment because more people might request their data.

Once your archive is on your machine, you will have a browsable HTML archive of your tweets, direct messages, and moments including media like images, videos, and GIFs. This is nice, but it also has a few flaws. For one, you can’t easily copy your Tweets somewhere else, for example, into your website because they are stored in a complex JSON structure. But even more dangerous: your links are all still t.co links. This hides the original URL you shared and redirects all traffic over Twitter’s servers. But this is not only inconvenient, it is also dangerous. Just imagine what happens when t.co ever goes down: all URLs you ever shared are now unretrievable. And then, there are the images: all images in your archive are much smaller than the ones you originally shared, and when you click to expand them, you are taken to the Twitter website once again. So, your Twitter archive is far from being a safe, independent backup in case you want to delete your account or if Twitter ever dies completely.

But luckily, there is help in the form of a Python script called Twitter Archive Parser, a project startet by Tim Hutton.

This script does the following:

  • It converts the tweets in your archive to markdown with embedded images, videos and links.
  • It replaces t.co URLs with their original versions.
  • It copies used images to an output folder, to allow them to be moved to a new home.
  • Afterwards, it asks if you want to try downloading the original size images using downloadbetterimages.py.

The most challenging part of using it might be that you need to install Python3 first (😅). But once this is done, all you have to do is to put the little parser.py script into your downloaded Twitter archive folder and run it via python parser.py.

And that’s it. My Tweets are now available as Markdown and all links are converted into the originally shared URLs. 🎉

Like in this Tweet from 2011:

'I've been doing this for fifty years and I know what I'm doing!' Rand v Jobs: when egos collide http://www.creativereview.co.uk/cr-blog/2011/october/jobs-v-rand #design

(Originally on Twitter: [Wed Nov 02 07:05:48 +0000 2011](https://twitter.com/m_ott/status/131628052932341761))

If you also want to download high-res versions of your images, you can optionally do that as well, but it will ask you first – and downloading them can take a little while.

There are other scripts that do similar or slightly other things – you can find links to some of them in the GitHub repository as well – yet in my case those two little scripts did the job well and it was all I needed (for now). Thanks a lot, Tim! 👏

~

464 Webmentions

Photo of @matthiasott
@matthiasott
@matthiasott Oh marvelous, thanks so much Matthias! I’ll give it a shot today.
Photo of @beep
@beep
@beep Sorry, I didn’t want to share my post, that was a copy & paste error . Here’s the direct link to the repo: https://github.com/timhutton/twitter-archive-parser GitHub - timhutton/twitter-archive-parser: Python code to parse a Twitter archive and output in various ways
Photo of dickelippe
dickelippe
Thank you for explicitly pointing out, that this script needs Python 3! I couldn't get the script to run yesterday and after checking the version found out that python 2.x _and_ 3.x was installed. Eventually was able to run it with python3 parser.py
Photo of @matthiasott
@matthiasott
Hey @matthiasott, I added Converting Your Twitter Archive to Markdown https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown to https://twittertomastodon.com/resources Great post! Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
Photo of Nelson Minar
Nelson Minar
Twitter archive improver: A Python tool to download images, replace t.co links with real links, etc matthiasott.com/notes/converti… + twitter archive socialmedia
Photo of Thomas Steiner
Thomas Steiner
Converting your Twitter Archive to Markdown, by @m_ott: matthiasott.com/notes/converti…. This might come in handy, who knows…
Photo of @kojote@sueden.social
@kojote@sueden.social
Twitter Parser running matthiasott.com/notes/converti… Don't forget to grab your archive and get it converted to markup + download the nicer images #twitter
Photo of @frank
@frank
@frank I actually have to run python3 parser.py instead of python parser.py Not a biggy, may differ on OS (use different *buntu's)
Photo of @matthiasott
@matthiasott
@frank @matthiasott thanks for the script recommendation. Especially converting short links to full link is very handy for archiving twitter feed
Photo of @frank
@frank
@frank https://anagora.org/Obsidian [[obsidian]] - anagora.org
Photo of dickelippe
dickelippe
Two weeks ago it took only 24hrs for mine, too. But I requested one on friday which somehow vanished before Bring available. And then I requested on on monday which is still not available. But I'm also waiting for my Facebook Data since monday...
Photo of zila
zila
t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
Photo of Flo
Flo
Same here. 🤔;
Photo of @matthiasott
@matthiasott
@matthiasott I’m hoping I can download mine, tried the other day and couldn’t because Twitter wouldn’t send an email code anymore
Photo of @matthiasott
@matthiasott
@frank @matthiasott Really nice ,i was looking for something similar! "The most challenging part of using it might be that you need to install Python3 first (😅;)." I hope only my poor English skill prevent me to get that this is a joke. Is it??
Photo of Milo
Milo
Mine took two days. Keep the faith. Btw, this could be nifty if you once get your archive. matthiasott.com/notes/converti…
Photo of barb dybwad
barb dybwad
P.P.S. How to convert your Twitter Archive to #Markdown matthiasott.com/notes/converti…
Photo of @matthiasott
@matthiasott
Go a step further and follow @matthiasott 's tutorial on how to convert the archive to markdown with high-res images. https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
Photo of Bianca Kastl
Bianca Kastl
Reminder: Sofern ihr Twitter Archive anfordert (sinnvoll, ja) ist es sinnvoll, die Daten in ein nutzbares Format zu konvertieren: Beschreibung z.B. unter matthiasott.com/notes/converti…
Photo of @matthiasott
@matthiasott
@matthiasott I requested my archive 3 days ago and haven't got it yet. Not confident it's coming. 😬;
Photo of @matthiasott
@matthiasott
@matthiasott Bookmarking for later… I just requested mine this morning, so fingers crossed it materialises before the whole thing is smoking ruins.
Photo of Karolina Wartolowska
Karolina Wartolowska
Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer matthiasott.com/notes/converti…
Photo of @matthiasott
@matthiasott
@matthiasott asked for mine days ago, was told notified when done, still nothing
Photo of Manuel Trujillo
Manuel Trujillo
t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
Photo of @timhutton
@timhutton
@matthiasott We should always caveat with don't blindly run scripts from the internet. Always know what your are running especially when downloading random stuff. E.g. I usually check the library imports at the top of the file for capabilities, though I trust @timhutton 😘; using import_module in a simple script to hot load libraries should raise a flag and bears further investigation. (It requests urls for ...
Photo of @matthiasott
@matthiasott
@nf3xn @matthiasott Definitely good advice! Re t.co links: we just use the expansion saved in the JSON, so no internet access. For retrieving full-res images we do access the internet, but we ask the user for confirmation first. Is there a nicer solution for beginners than hot loading uninstalled packages? We ask for confirmation on each one.
Photo of @matthiasott
@matthiasott
@AshleyMarineP @matthiasott Mine took like 3 days, but I eventually got it.
Photo of @matthiasott
@matthiasott
@matthiasott What a fascinating job you've done with that Twitter-to-Markdown script, Matthias! I'm just half literate when using command prompt, and I've been able to follow every step anyway! Thank you, thank you, thank you!
Photo of @timhutton
@timhutton
@matthiasott @timhutton Oops! Sorry! I'm still finding my feet around here and didn't realise. Of course it was Tim from GitHub. Thank you so much, Tim!! And thanks, Matthias, for sharing this wonderful tip!
Photo of @timhutton
@timhutton
@cantavest Hi Rosanna! Thank you – but I only wrote down how it worked for me, all the credit for this amazing little script goes to @timhutton !!! 🙌;😊;
Photo of Flo
Flo
Du weißt doch … das Beste kommt zum Schluss! 😉;
Photo of dickelippe
dickelippe
Ich habe jetzt mal für einen meiner anderen Accounts das Archiv angefordert. Mal sehen, ob das da schneller geht. Der Account hat nur ca. 50 Tweets und keine Follower etc...
Photo of Pablo Lara H
Pablo Lara H
🐦; Converting Your Twitter Archive to Markdown by Matthias Ott @m_ott @matthiasott@mastodon.social #TwitterArchive #TwitterToMarkdown matthiasott.com/notes/converti…
Photo of Lunks
Lunks
t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
Photo of @matthiasott
@matthiasott
@andy I turned mine into Markdown and put it in Obsidian. https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown Via @matthiasott Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
Photo of Rafael (en @exosferablog@scholar.social)
Rafael (en @exosferablog@scholar.social)
Una vez hecho os recomiendo correr el siguiente script de python para eiminar cualquier dependencia de los servidores y bajar archivos adjuntos originales matthiasott.com/notes/converti… (por eso ahora mismo no la tengo en privado por que está leyendo todos mis tuits el script)
Photo of Fulgencio Sanmartín
Fulgencio Sanmartín
And as sustainable and #digipres follow-up so you may read and reuse it: convert that eArchive to markdown to recover your original links and media: matthiasott.com/notes/converti… by @m_ott
Photo of Kimberly M. Bonner NOW On POST
Kimberly M. Bonner NOW On POST
Get all your Twitter posts bf she blows. Great article that helps with this process. Converting Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti…
Photo of Barbara OBrien
Barbara OBrien
Converting Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti…
Photo of @roland@giersig.net ????;
@roland@giersig.net ????;
t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
Photo of Stephen Downes
Stephen Downes

Converting Your Twitter Archive to Markdown

Today's Mastodon/Twitter update. Matthias Ott explains how to convert your Twitter archive into something useful (including fixing all the links, which because they're t.co links in the archive, will break when Twitter fails. From Jisc's Steve Bailey, things to consider when setting up your Mastodon instance. Miguel Guhlin with some brief reflections on transitioning from 13K followers to zero. Fediverse.info ...
Photo of Reda Sadki
Reda Sadki
Save your tweets matthiasott.com/notes/converti…
Photo of Jacky
Jacky
Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer matthiasott.com/notes/converti…
Photo of Ross of Ottawa
Ross of Ottawa
Requesting my #TwitterArchive. It's not so much that I need it or am likely to dig through it, but there are so many #python tools to process it popping up that I'd like to try some of them. This article is interesting… matthiasott.com/notes/converti…
Photo of Vivienne Dunstan
Vivienne Dunstan
The reason why I finally installed Python3 on my new Mac was so I could expand the Twitter archive I’d downloaded, replacing t.co aliases (won’t work if Twitter goes belly up) and downloading full images. Here’s info on how to do it. matthiasott.com/notes/converti…
Photo of ivo
ivo
Iemand vinden die het volgende Python script op je Twitter archief uitvoert: matthiasott.com/notes/converti… Hiermee worden de t.co urls vervangen door de originele urls. Natuurlijk wel doen voordat de t.co service (ooit) uit de is 😉;
Photo of Richard Earney
Richard Earney
Have you seen the python script to make it offline and in Markdown? matthiasott.com/notes/converti…
Photo of https://mastodon.social/@kylemackie
https://mastodon.social/@kylemackie
1) request your Twitter archive, and then 2) Convert Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti… - seems scary at first, but just follow the steps. Thank you, Elon for presenting me with the opportunity to learn how to run python scripts.
Photo of Denny
Denny
"The bird is not well. So it is time to request and download an archive of your Twitter data now, if you haven’t done this recently..." matthiasott.com/notes/converti…
Photo of Kathy Schrock
Kathy Schrock
Used the directions in this article to convert my Twitter archive to Markdown. matthiasott.com/notes/converti… Found some fun photos! These were my roomies at a DEN Summer Institute in Nashville in 2014! #findyourtribe
Photo of https://toot.cat/@zeenix
https://toot.cat/@zeenix
Seems like Mastodon (despite a few limitations) is much better for my mental health than this platform. So most likely I'll be deactivating my account here soon. P.S. I've already archived all my twitter data using this guide: matthiasott.com/notes/converti…
Photo of Mariana
Mariana
Not sure where I saw this but you might be able to fix the links. matthiasott.com/notes/converti…
Photo of Ross of Ottawa
Ross of Ottawa
This link is a helpful one, with info on downloading archive, and processing it. matthiasott.com/notes/converti…
Photo of DK
DK
USE this python code to convert all your tweets into Markdown whilst converting the shortened urls to the originals plus downloads all associated images github.com/timhutton/twit… with thanks to @_tim_hutton_ & @m_ott matthiasott.com/notes/converti… (worked for me)

Likes

Reposts