Con­vert­ing Your Twit­ter Archive to Markdown

Posted by

470 Webmentions

The bird is not well. So it is time to request and download an archive of your Twitter data now, if you haven’t done this recently. After you have requested your archive, it can take a while until you receive it. I requested my archive on the day Elon walked in and got an email notice that my archive is ready for download after about a day. I suppose it might take a little longer at the moment because more people might request their data.

Once your archive is on your machine, you will have a browsable HTML archive of your tweets, direct messages, and moments including media like images, videos, and GIFs. This is nice, but it also has a few flaws. For one, you can’t easily copy your Tweets somewhere else, for example, into your website because they are stored in a complex JSON structure. But even more dangerous: your links are all still t.co links. This hides the original URL you shared and redirects all traffic over Twitter’s servers. But this is not only inconvenient, it is also dangerous. Just imagine what happens when t.co ever goes down: all URLs you ever shared are now unretrievable. And then, there are the images: all images in your archive are much smaller than the ones you originally shared, and when you click to expand them, you are taken to the Twitter website once again. So, your Twitter archive is far from being a safe, independent backup in case you want to delete your account or if Twitter ever dies completely.

But luckily, there is help in the form of a Python script called Twitter Archive Parser, a project startet by Tim Hutton.

This script does the following:

  • It converts the tweets in your archive to markdown with embedded images, videos and links.
  • It replaces t.co URLs with their original versions.
  • It copies used images to an output folder, to allow them to be moved to a new home.
  • Afterwards, it asks if you want to try downloading the original size images using download_better_images.py.

The most challenging part of using it might be that you need to install Python3 first (😅). But once this is done, all you have to do is to put the little parser.py script into your downloaded Twitter archive folder and run it via python parser.py.

And that’s it. My Tweets are now available as Markdown and all links are converted into the originally shared URLs. 🎉

Like in this Tweet from 2011:

'I've been doing this for fifty years and I know what I'm doing!' Rand v Jobs: when egos collide http://www.creativereview.co.uk/cr-blog/2011/october/jobs-v-rand #design

(Originally on Twitter: [Wed Nov 02 07:05:48 +0000 2011](https://twitter.com/m_ott/status/131628052932341761))

If you also want to download high-res versions of your images, you can optionally do that as well, but it will ask you first – and downloading them can take a little while.

There are other scripts that do similar or slightly other things – you can find links to some of them in the GitHub repository as well – yet in my case those two little scripts did the job well and it was all I needed (for now). Thanks a lot, Tim! 👏

~

470 Webmentions

  3. Photo of Lumea SEO PPC Lumea SEO PPC
    Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer matthiasott.com/notes/converti… This script does the following: It converts the tweets in your archive to markdown with embedded images, videos and links. ...
  4. Photo of Michael Cox - mastodon.scot/@shinMC Michael Cox - mastodon.scot/@shinMC
    Locking down my account before final Mastodon move. Some useful links, if you're considering same: Archive & lockdown guide: nytimes.com/wirecutter/blo… Get a better version of your twitter archive: matthiasott.com/notes/converti… Best follower finder I've used: movetodon.org
  5. Photo of ????;LillyinWick ????;LillyinWick
    It's probably worth reading this if you've downloaded your twitter archive and are wondering what to do with it: matthiasott.com/notes/converti…
  6. Photo of DK DK
    USE this python code to convert all your tweets into Markdown whilst converting the shortened urls to the originals plus downloads all associated images github.com/timhutton/twit… with thanks to @_tim_hutton_ & @m_ott matthiasott.com/notes/converti… (worked for me)
  8. Photo of Ross of Ottawa Ross of Ottawa
    This link is a helpful one, with info on downloading archive, and processing it. matthiasott.com/notes/converti…
  9. Photo of Mariana Mariana
    Not sure where I saw this but you might be able to fix the links. matthiasott.com/notes/converti…
  10. Photo of https://toot.cat/@zeenix https://toot.cat/@zeenix
    Seems like Mastodon (despite a few limitations) is much better for my mental health than this platform. So most likely I'll be deactivating my account here soon. P.S. I've already archived all my twitter data using this guide: matthiasott.com/notes/converti…
  11. Photo of Kathy Schrock Kathy Schrock
    Used the directions in this article to convert my Twitter archive to Markdown. matthiasott.com/notes/converti… Found some fun photos! These were my roomies at a DEN Summer Institute in Nashville in 2014! #findyourtribe
  12. Photo of Denny Denny
    "The bird is not well. So it is time to request and download an archive of your Twitter data now, if you haven’t done this recently..." matthiasott.com/notes/converti…
  13. Photo of https://mastodon.social/@kylemackie https://mastodon.social/@kylemackie
    1) request your Twitter archive, and then 2) Convert Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti… - seems scary at first, but just follow the steps. Thank you, Elon for presenting me with the opportunity to learn how to run python scripts.
  14. Photo of Richard Earney Richard Earney
    Have you seen the python script to make it offline and in Markdown? matthiasott.com/notes/converti…
  15. Photo of ivo ivo
    Iemand vinden die het volgende Python script op je Twitter archief uitvoert: matthiasott.com/notes/converti… Hiermee worden de t.co urls vervangen door de originele urls. Natuurlijk wel doen voordat de t.co service (ooit) uit de is 😉
  16. Photo of Vivienne Dunstan Vivienne Dunstan
    The reason why I finally installed Python3 on my new Mac was so I could expand the Twitter archive I’d downloaded, replacing t.co aliases (won’t work if Twitter goes belly up) and downloading full images. Here’s info on how to do it. matthiasott.com/notes/converti…
  17. Photo of Ross of Ottawa Ross of Ottawa
    Requesting my #TwitterArchive. It's not so much that I need it or am likely to dig through it, but there are so many #python tools to process it popping up that I'd like to try some of them. This article is interesting… matthiasott.com/notes/converti…
  19. Photo of Jacky Jacky
    Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer matthiasott.com/notes/converti…
  22. Photo of Stephen Downes Stephen Downes

    Converting Your Twitter Archive to Markdown
    Today's Mastodon/Twitter update. Matthias Ott explains how to convert your Twitter archive into something useful (including fixing all the links, which because they're t.co links in the archive, will break when Twitter fails. From Jisc's Steve Bailey, things to consider when setting up your Mastodon instance. Miguel Guhlin with some brief reflections on transitioning from 13K followers to zero. Fediverse.info ...
  24. Photo of Barbara OBrien Barbara OBrien
    Converting Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti…
  25. Photo of Kimberly M. Bonner NOW On POST Kimberly M. Bonner NOW On POST
    Get all your Twitter posts bf she blows. Great article that helps with this process. Converting Your Twitter Archive to Markdown, by @m_ott matthiasott.com/notes/converti…
  26. Photo of Fulgencio Sanmartín Fulgencio Sanmartín
    And as sustainable and #digipres follow-up so you may read and reuse it: convert that eArchive to markdown to recover your original links and media: matthiasott.com/notes/converti… by @m_ott
  27. Photo of Rafael (en @exosferablog@scholar.social) Rafael (en @exosferablog@scholar.social)
    Una vez hecho os recomiendo correr el siguiente script de python para eiminar cualquier dependencia de los servidores y bajar archivos adjuntos originales matthiasott.com/notes/converti… (por eso ahora mismo no la tengo en privado por que está leyendo todos mis tuits el script)
  28. Photo of @matthiasott @matthiasott
    @andy I turned mine into Markdown and put it in Obsidian. https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown Via @matthiasott Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
  31. Photo of Lunks Lunks
    t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
  33. Photo of Pablo Lara H Pablo Lara H
    🐦 Converting Your Twitter Archive to Markdown by Matthias Ott @m_ott @matthiasott@mastodon.social #TwitterArchive #TwitterToMarkdown matthiasott.com/notes/converti…
  34. Photo of dickelippe dickelippe
    Ich habe jetzt mal für einen meiner anderen Accounts das Archiv angefordert. Mal sehen, ob das da schneller geht. Der Account hat nur ca. 50 Tweets und keine Follower etc...
  35. Photo of Flo Flo
    Du weißt doch … das Beste kommt zum Schluss! 😉
  38. Photo of @timhutton @timhutton
    @cantavest Hi Rosanna! Thank you – but I only wrote down how it worked for me, all the credit for this amazing little script goes to @timhutton !!! 🙌????;
  39. Photo of @timhutton @timhutton
    @matthiasott @timhutton Oops! Sorry! I'm still finding my feet around here and didn't realise. Of course it was Tim from GitHub. Thank you so much, Tim!! And thanks, Matthias, for sharing this wonderful tip!
  40. Photo of @matthiasott @matthiasott
    @matthiasott What a fascinating job you've done with that Twitter-to-Markdown script, Matthias! I'm just half literate when using command prompt, and I've been able to follow every step anyway! Thank you, thank you, thank you!
  42. Photo of @matthiasott @matthiasott
    @AshleyMarineP @matthiasott Mine took like 3 days, but I eventually got it.
  43. Photo of @matthiasott @matthiasott
    @nf3xn @matthiasott Definitely good advice! Re t.co links: we just use the expansion saved in the JSON, so no internet access. For retrieving full-res images we do access the internet, but we ask the user for confirmation first. Is there a nicer solution for beginners than hot loading uninstalled packages? We ask for confirmation on each one.
  44. Photo of @timhutton @timhutton
    @matthiasott We should always caveat with don't blindly run scripts from the internet. Always know what your are running especially when downloading random stuff. E.g. I usually check the library imports at the top of the file for capabilities, though I trust @timhutton 😘 using import_module in a simple script to hot load libraries should raise a flag and bears further investigation. (It requests urls for ...
  46. Photo of Manuel Trujillo Manuel Trujillo
    t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
  47. Photo of @matthiasott @matthiasott
    @matthiasott asked for mine days ago, was told notified when done, still nothing
  48. Photo of Karolina Wartolowska Karolina Wartolowska
    Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer matthiasott.com/notes/converti…
  51. Photo of @matthiasott @matthiasott
    @matthiasott Bookmarking for later… I just requested mine this morning, so fingers crossed it materialises before the whole thing is smoking ruins.
  53. Photo of @matthiasott @matthiasott
    @matthiasott I requested my archive 3 days ago and haven't got it yet. Not confident it's coming. 😬
  56. Photo of Bianca Kastl Bianca Kastl
    Reminder: Sofern ihr Twitter Archive anfordert (sinnvoll, ja) ist es sinnvoll, die Daten in ein nutzbares Format zu konvertieren: Beschreibung z.B. unter matthiasott.com/notes/converti…
  58. Photo of Ercüment Ercüment
    Bu iş;e yarayabilir belki, çalı;ş;ı;yor mu henüz denemedim matthiasott.com/notes/converti…
  59. Photo of @matthiasott @matthiasott
    Go a step further and follow @matthiasott 's tutorial on how to convert the archive to markdown with high-res images. https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
  60. Photo of barb dybwad barb dybwad
    P.P.S. How to convert your Twitter Archive to #Markdown matthiasott.com/notes/converti…
  61. Photo of Milo Milo
    Mine took two days. Keep the faith. Btw, this could be nifty if you once get your archive. matthiasott.com/notes/converti…
  62. Photo of @matthiasott @matthiasott
    @frank @matthiasott Really nice ,i was looking for something similar! "The most challenging part of using it might be that you need to install Python3 first (😅)." I hope only my poor English skill prevent me to get that this is a joke. Is it??
  63. Photo of @matthiasott @matthiasott
    @matthiasott I’m hoping I can download mine, tried the other day and couldn’t because Twitter wouldn’t send an email code anymore
  65. Photo of zila zila
    t.co links. Here’s a quick how-to: matthiasott.com/notes/converti… mastodon.social/@matthiasott/1… (2/2)
  68. Photo of dickelippe dickelippe
    Two weeks ago it took only 24hrs for mine, too. But I requested one on friday which somehow vanished before Bring available. And then I requested on on monday which is still not available. But I'm also waiting for my Facebook Data since monday...
  69. Photo of @frank @frank
    @frank https://anagora.org/Obsidian [[obsidian]] - anagora.org
  70. Photo of @matthiasott @matthiasott
    @frank @matthiasott thanks for the script recommendation. Especially converting short links to full link is very handy for archiving twitter feed
  71. Photo of @frank @frank
    @frank I actually have to run python3 parser.py instead of python parser.py Not a biggy, may differ on OS (use different *buntu's)
  73. Photo of @kojote@sueden.social @kojote@sueden.social
    Twitter Parser running matthiasott.com/notes/converti… Don't forget to grab your archive and get it converted to markup + download the nicer images #twitter
  74. Photo of Thomas Steiner Thomas Steiner
    Converting your Twitter Archive to Markdown, by @m_ott: matthiasott.com/notes/converti…. This might come in handy, who knows…
  75. Photo of Nelson Minar Nelson Minar
    Twitter archive improver: A Python tool to download images, replace t.co links with real links, etc matthiasott.com/notes/converti… + twitter archive socialmedia
  79. Photo of @matthiasott @matthiasott
    Hey @matthiasott, I added Converting Your Twitter Archive to Markdown https://matthiasott.com/notes/converting-your-twitter-archive-to-markdown to https://twittertomastodon.com/resources Great post! Converting Your Twitter Archive to Markdown · Matthias Ott – User Experience Designer
  80. Photo of dickelippe dickelippe
    Thank you for explicitly pointing out, that this script needs Python 3! I couldn't get the script to run yesterday and after checking the version found out that python 2.x _and_ 3.x was installed. Eventually was able to run it with python3 parser.py
  81. Photo of @beep @beep
    @beep Sorry, I didn’t want to share my post, that was a copy & paste error . Here’s the direct link to the repo: https://github.com/timhutton/twitter-archive-parser GitHub - timhutton/twitter-archive-parser: Python code to parse a Twitter archive and output in various ways
  82. Photo of @matthiasott @matthiasott
    @matthiasott Oh marvelous, thanks so much Matthias! I’ll give it a shot today.

166 Reposts

ⓘ Webmentions are a way to notify other websites when you link to them, and to receive notifications when others link to you. Learn more about Webmentions.

More Notes