Tools for automated transcription of audio and video fragments

Speech recognition, speech-to-text (STT) or automated speech recognition (ASR) is a technology that makes it possible to convert spoken text in videos or audio into text, such as the automatic subtitles on YouTube or Zoom. In this overview we focus on transcribing interviews for oral history, but virtual assistants like Siri or Google Assistant are also a form of this technology.

Speech recognition is a relatively old technology. The first commercial tools appeared in the early 1990s. They use models, systems that are trained on a certain set of data to recognize patterns and make decisions without human intervention. Speech recognition models are language models trained on audio such as interviews, audiobooks, lectures and presentations. The strength of the speech recognition tool depends enormously on the model used.

Possibilities of speech recognition tools

A research by meemoo in 2020-2021 showed that speech recognition technology was less good at transcribing audio from conversations, but great strides have been made in recent years which has greatly improved the technology, partly due to the greater computing power of computers, the progress in technology related to machine learning and big data and the improved language models. This has ensured that the tools generate more natural language and produce fewer nonsensical texts, thus improving transcriptions of conversations enormously.

In addition, speech recognition tools can do more than just transcribe text. They can also:

• recognize different speakers and indicate which text was spoken by which speaker
• indicate and remove filler words such as euhm from the text
• indicate silences
• create summaries and translations
• ...

Several tools also have a possibility to add a dictionary. Here you can set specialized words that would otherwise be transcribed incorrectly.

Points of attention when using the tools

You can take into account a number of factors to achieve better results:

• The strength of the speech recognition tools depends on the model used: e.g., which languages does the model support and how well can the model handle dialects or colloquial language? The choice for the model depends on the needs and use cases, such as the trade-off between speed and accuracy, and the language that must be transcribed. In the overview we chose accuracy (it is important that the text is correct than that the tool is fast) and Dutch as colloquial language. They were also all tested on interviews in which slight dialect or intermediate language was spoken. The most accurate tools scored well for this, but generally it was noted that interviews with pronounced dialects resulted in poorer transcriptions.
• Another important point of attention is the quality of the recording. Recordings with clear sound and without background noise give better results than recordings with poor sound quality (e.g. noise) and background noise. Speakers who speak clearly are also transcribed better than speakers who mumble.
• Desktop applications use the computing power of the computer when transcribing the text. If your computer has a dedicated GPU, the transcription will proceed much faster. A dedicated GPU is a special GPU with its own card connected to the motherboard, while an integrated GPU is embedded in the same chip as the CPU. This article (in Dutch) explains more about the different GPUs. If your computer does not have this GPU, then the CPU or processor is used. This is slower. You can also use an online service so that you are not limited by the limits of your computer.

Overview of the different tools

At the request of heritage organisations for support in writing out interviews, a number of tools were tested. Most tools can do more than just transcribe, but those functions were not investigated deeper.

Amberscript

Amberscript is a commercial web platform that allows you to create transcriptions and subtitles for audio and video. When you want to transcribe audio or video, you must upload those fragments into their web environment. Which model is used is not documented.

It supports:

• different languages, including Dutch;
• speaker identification;
• and indication of timestamps

Advantages:

• Editing transcriptions is very clear.
• Possibility to add a dictionary.

Disadvantages:

• Speaker identification is inaccurate when there are pauses in the audio fragment.
• Pauses are not indicated.
• Accuracy in Dutch is not good. The transcribed text contains quite a few errors.

Using Amberscript costs money. If you want to pay once, it will cost you €15/hour, whereby you can purchase a maximum of hunderd hours at a time. It is also possible to take out a subscription. You then pay €40/month per five hours of uploaded audio or video if you take a yearly subscription, or €50/month for five hours of audio or video. It is not possible to transfer unused hours to another month.

Audapolis

Audapolis is an open source and offline desktop application that uses the Vosk model. Vosk is a relatively small, but also older model that is mainly used for chatbots, smart home applications and virtual assistants. It was originally developed for smaller devices such as smartphones and microcomputers.

The tool can transcribe twenty languages, including Dutch, identifies which text was spoken by which speaker and also indicates timestamps. Because the tool works offline, you can also use the tool, for example, on the train when traveling home after an interview.

Advantages:

• free and open source application;
• works offline;
• includes a built-in editor to improve the transcribed text;
• detects and removes euhms and pauses.

Disadvantages:

• does not work well with accents or (slight) dialects;
• accuracy of transcribed text and speaker identification is low;
• text can only be exported in HTML format;
• tool hasn't been updated for a year.

Limecraft

Limecraft is an online platform for managing audio and video and offers the possibility to transcribe that content as an extra feature. It uses six models for this: Vocapia, Speechmatics, Google Speech, Microsoft Auze, Scriptix and Kaldi. As a user you cannot choose which of those models you want to use. The platform makes that choice for you based on the purposes and the language you choose. Because so many models are used, it can transcribe more than hunderd languages, including Dutch. Just like the previous tools, it can identify speakers and indicate timestamps. On the platform, multiple people can collaborate on a transcription, like working together on a document in Sharepoint or Google Drive.

Advantages:

• user-friendly interface with simple editing possibilities;
• fast;
• extensive export options (.pdf, .csv, .doc);
• possibility to add your own dictionary;
• has extra features, such as creating subtitles, topic detection and making summaries;
• it is a Belgian company, which means support and contact are in Dutch and the same time zone.

Disadvantages:

• Uhms are not recognized well and silences are not indicated.
• The distinction between speakers is not always good, but you can edit this.
• It makes strange things of words that it does not know.
• Transcription is an extra feature, making the platform quite expensive and having many functionalities you do not need if you only want to transcribe.

The prices range from free (one user with 5h of material), €85/month (five users with 25h of material) or €275/month (for larger teams). To have audio and video transcribed, you pay an extra €15/hour. Limecraft also offers the possibility to translate that transcription. That also costs €15/hour.

Sonix

Sonix is also a commercial web platform where you can collaborate on transcriptions. There are extensive possibilities to edit transcriptions, adjust timestamps, etc. It can transcribe more than 49 languages, including Dutch, recognizes the different speakers well and indicates timestamps. Finally, it also has the ability to indicate in color codes how certain Sonix is about certain transcriptions.

Advantages:

• user-friendly interface with extensive and simple editing possibilities;
• transcribes quickly;
• possibility to add your own dictionary for specific words;
• extensive export options
• has an extra (paid) feature to create summaries of transcriptions

Disadvantages:

• Uhms are not recognized well.
• It compresses the original media files when using the cheapest tariff plan, so you can no longer export the original media files.
• With the cheapest tariff plan you only have support per email.

Sonix has different subscription models:

• Standard pay-as-you-go where you pay $10/hour audio or video
• Premium for organizations that regularly want to transcribe audio and video and need more collaboration possibilities. For this you pay $5/hour audio or video and monthly $22 per user.
• Enterprise for high volumes of transcription needs with extensive collaboration options and content analysis.

You can test the possibilities of Sonix free of charge for 30 minutes of audio and video.

.

.

Speechmatics

Speechmatics is a company that has developed its own closed speech recognition model and offers APIs and a platform for a fee to transcribe and translate audio and video. They try to compete with large companies such as Google, Amazon and Microsoft and according to tests their model scores better than those tech companies. It can be used for both recorded media and for real-time audio and video. The software can transcribe 52 languages, including Dutch, identify speakers and indicate timestamps. Speechmatics focuses solely on transcription. Media files and their transcriptions are therefore only stored on the platform for one week. This has the advantage that they are one of the cheapest speech-to-text providers.

Advantages:

• very accurate;
• removes uhms;
• platform focuses solely on transcription, so you don’t have to pay for unnecessary bells and whistles;
• exports to plain text (.txt), SRT (for subtitles) and JSON.

Disadvantages:

• hallucinates on terms it doesn’t know;
• does not indicate silences;
• no timestamps when exporting to plain text or when using the copy function;
• media files and transcriptions are only stored for one week on the web platform;
• the web platform has difficulty uploading video.

Speechmatics does not have a subscription formula. You can get 4h of uploaded audio or video transcribed free every month (and also 4h of real-time audio and video). In addition, you pay per hour and the price depends on the desired accuracy of the transcribed text. You pay $0.8/hour for standard accuracy and $1.04/hour for the enhanced accuracy or most accurate model.

.

.

noScribe

NoScribe is a free open source tool for transcribing audio and video. It’s an offline desktop application that uses the Whisper model (for more info about Whisper, see below) from OpenAI, the company that also developed ChatGTP. NoScribe can transcribe more than 99 languages, including Dutch, identifies speakers and provides timestamps. It doesn't yet use the most recent (and most accurate for Dutch) model of Whisper because that version of that model scores less well in some other languages.

Advantages:

• free and open source;
• very accurate; hallucinates less and remains consistent with terms it doesn’t know;
• has editor software to improve transcriptions;
• can export to HTML, plain text (.txt) and VTT (for subtitles);
• can use a dedicated GPU to make the transcription faster.

Disadvantages:

• speed of transcription depends on your own computer;
• can hallucinate on silence, although we couldn't confirm this in practice;
• multilingual audio (e.g., an interview where different languages are spoken) is not supported;
• sometimes makes mistakes when recognizing speakers.

Read the manual of noScribe here.

Whisper

Whisper is een model voor spraakherkenning, ontwikkeld door OpenAI, dat voor het eerst als opensourcesoftware uitgebracht werd in 2022. Het kan gebruikt worden om verschillende talen te transcriberen en om verschillende talen te laten vertalen naar Engels. Het is ingebouwd in verschillende spraakherkenningstools, zoals noScribe, maar kan ook gebruikt worden als commandlinetool. Whisper kan honderd talen transcriberen, waaronder Nederlands, en duidt tijdscodes aan. Uit testen op twee datasets blijkt ook dat de laatste versie van Whisper zeer goed scoort op Nederlands.

Voordelen:

• opensource en gratis
• zeer accuraat; de commandlinetool gebruikt het meest recentste model, wat voor het Nederlands ook het meest accurate model is
• exporst zijn mogelijk in platte tekst (.txt), SRT (ondertiteling), VTT (ondertiteling), TSV (een tabulair formaat vergelijkbaar met CSV) en JSON

Nadelen:

• snelheid is afhankelijk van de eigen computer, zeker als je geen dedicated GPU hebt (zie noScribe) kan de transcriptie zeer langzaam gaan (maar nog steeds sneller dan als je het zelf zou doen)
• duidt geen stiltes aan
• Whisper kan hallucineren op stilte, maar het is mogelijk om via de commandline hier aanpassingen aan te doen
• meertalige audio is niet ondersteund
• geen intuïtieve grafische interface (GUI); enkel te gebruiken via de command line.
• geen omgeving waarin je de transcriptie kan verbeteren.

Conclusie

Er zijn, afhankelijk van je noden, verschillende tools die je kan gebruiken voor het geautomatsieerd transcriberen van audio- en videofragmenten. Om eenvoudiger een keuze te maken, kan je onderstaande tabel gebruiken. In de tabel wordt onder meer aangeduid welke mogelijkheden de tools volledig (aangeduid met X) of gedeeltelijk (aangeduid met /) ondersteunen en hun prijscategorie: €0 betekent gratis, € staat voor een prijs minder dan €5/uur, €€ geeft een prijs tussen €5/uur en €15/uur weer en €€€ is een prijs hoger dan €15/uur.

	Amberscript	Audapolis	Limecraft	Sonix	Speechmatics	noScribe	Whisper
Ondersteunt Nederlands	X	X	X	X	X	X	X
Accuraat			X	X	X	X	X
Gebruiksvriendelijk	X	X	X	X	X	X
Snel	X		X	X	X
Mogelijkheid tot verbeteren van transcriptie	X	X	X	X		X
Mogelijkheid tot samenwerken aan transcriptie			X	X
Identificeert sprekers	X	X	X	X	X	X
Detecteert euhms		X		/	X	X	X
Duidt tijdscodes aan	X		X	X	/	X	X
Detecteert pauzes	X	X		X		X	X
Mogelijkheid tot toevoegen eigen woorden	X			X
Exportformaten	.csv, .doc, .json, .rtf, .srt, .stl, .txt, .vtt	.html	.docx, .pdf, .srt, .txt, .vtt	.csv, .doc, .pdf	.json, .srt, .txt	.html, .txt, .vtt	.csv, .json, .srt, .tsv, .txt, .vtt
Open source		X				X	X
Cloudservice	X		X	X	X
Prijs	€€	€0	€€€	€€	€	€0	€0

Auteur: Nastasia Vanderperren (Meemoo, Vlaams instituut voor het archief) en Lode Scheers (Meemoo, Vlaams instituut voor het archief)