The sections below give objective criteria for evaluating the usability of machine translation software output. == Stationarity or canonical form == Do repeated translations converge on a single expression in both languages? I.e. does the translation method show stationarity or produce a canonical form? Does the translation become stationary without losing the original meaning? This metric has been criticized as not being well correlated with BLEU (BiLingual Evaluation Understudy) scores. == Adaptive to colloquialism, argot or slang == Is the system adaptive to colloquialism, argot or slang? The French language has many rules for creating words in the speech and writing of popular culture. Two such rules are: (a) The reverse spelling of words such as femme to meuf. (This is called verlan.) (b) The attachment of the suffix -ard to a noun or verb to form a proper noun. For example, the noun faluche means "student hat". The word faluchard formed from faluche colloquially can mean, depending on context, "a group of students", "a gathering of students" and "behavior typical of a student". The Google translator as of 28 December 2006 doesn't derive the constructed words as for example from rule (b), as shown here: Il y a une chorale falucharde mercredi, venez nombreux, les faluchards chantent des paillardes! ==> There is a choral society falucharde Wednesday, come many, the faluchards sing loose-living women! French argot has three levels of usage: familier or friendly, acceptable among friends, family and peers but not at work grossier or swear words, acceptable among friends and peers but not at work or in family verlan or ghetto slang, acceptable among lower classes but not among middle or upper classes The United States National Institute of Standards and Technology conducts annual evaluations [1] Archived 2009-03-22 at the Wayback Machine of machine translation systems based on the BLEU-4 criterion [2]. A combined method called IQmt which incorporates BLEU and additional metrics NIST, GTM, ROUGE and METEOR has been implemented by Gimenez and Amigo [3]. == Well-formed output == Is the output grammatical or well-formed in the target language? Using an interlingua should be helpful in this regard, because with a fixed interlingua one should be able to write a grammatical mapping to the target language from the interlingua. Consider the following Arabic language input and English language translation result from the Google translator as of 27 December 2006 [4]. This Google translator output doesn't parse using a reasonable English grammar: وعن حوادث التدافع عند شعيرة رمي الجمرات -التي كثيرا ما يسقط فيها العديد من الضحايا- أشار الأمير نايف إلى إدخال "تحسينات كثيرة في جسر الجمرات ستمنع بإذن الله حدوث أي تزاحم". ==> And incidents at the push Carbuncles-throwing ritual, which often fall where many of the victims - Prince Nayef pointed to the introduction of "many improvements in bridge Carbuncles God would stop the occurrence of any competing." == Semantics preservation == Do repeated re-translations preserve the semantics of the original sentence? For example, consider the following English input passed multiple times into and out of French using the Google translator as of 27 December 2006: Better a day earlier than a day late. ==> Améliorer un jour plus tôt qu'un jour tard. ==> To improve one day earlier than a day late. ==> Pour améliorer un jour plus tôt qu'un jour tard. ==> To improve one day earlier than a day late. As noted above and in, this kind of round-trip translation is a very unreliable method of evaluation. == Trustworthiness and security == An interesting peculiarity of Google Translate as of 24 January 2008 (corrected as of 25 January 2008) is the following result when translating from English to Spanish, which shows an embedded joke in the English-Spanish dictionary which has some added poignancy given recent events: Heath Ledger is dead ==> Tom Cruise está muerto This raises the issue of trustworthiness when relying on a machine translation system embedded in a Life-critical system in which the translation system has input to a Safety Critical Decision Making process. Conjointly it raises the issue of whether in a given use the software of the machine translation system is safe from hackers. It is not known whether this feature of Google Translate was the result of a joke/hack or perhaps an unintended consequence of the use of a method such as statistical machine translation. Reporters from CNET Networks asked Google for an explanation on January 24, 2008; Google said only that it was an "internal issue with Google Translate". The mistranslation was the subject of much hilarity and speculation on the Internet. If it is an unintended consequence of the use of a method such as statistical machine translation, and not a joke/hack, then this event is a demonstration of a potential source of critical unreliability in the statistical machine translation method. In human translations, in particular on the part of interpreters, selectivity on the part of the translator in performing a translation is often commented on when one of the two parties being served by the interpreter knows both languages. This leads to the issue of whether a particular translation could be considered verifiable. In this case, a converging round-trip translation would be a kind of verification.
Invoicera
Invoicera is an online invoicing software. The software was created by a company with the same name that was founded in 2006, had 125 employees, and is based in India. It allows users to monitor, dispatch, and accept invoices in one web service. After signing up for the service, users are assigned a personal subdomain to set up their invoice configuration. It allows users to add clients' data to the service through uploading a Microsoft Excel file. Invoicera is compatible with businesses of varying sizes, including freelancers, small businesses, and large businesses. It is compatible with Basecamp, a project-management tool, so Invoicera can upload data from Basecamp. The software interfaces with more than 25 payment gateways. It supports subscriptions and repeated invoices and allows clients to schedule late fees when payments have not been made on time. Invoicera uses freemium model, letting users dispatch an unrestricted number of invoices to at most three customers. Chelsea Krause wrote in a 2019 review for Merchant Maverick, "Unfortunately, the software isn't as developed as it could be. Time tracking and reporting are limited and there are no live bank feeds — which is surprising for a company so focused on automation (especially since even many of the worst invoicing options out there still offer live bank feeds)." She further criticized Invoicera for having bad customer service and the software for not having recent changes. Brian Turner wrote in TechRadar that Invoicera had fewer templates compared to the other services he reviewed but "the ones offered are fully customizable". Rob Clymo wrote in TechRadar that "Invoicera lets you automate your invoicing and billing needs without too much in the way of hassle" and that although it "isn't a complete accounts solution ... it's a powerful supplement".
Boba liberal
Boba liberal is a term mostly used within the Asian diaspora communities in the West, especially in the United States. It describes someone of East or Southeast Asian descent living in the West who has a shallow, surface-level liberal outlook. It is also occasionally used to describe conservatives who weaponize their East or Southeast Asian identity. The neologism emerged among the Asian American leftist community on Twitter who accused "boba liberals" of only holding their liberal beliefs to appear more white-adjacent by engaging in progressive social movements or viewpoints, while at the same time disregarding and trivializing issues concerning Asians. Mary Chao, writing for The North Jersey Record, said that "Asians call peers boba liberals when they aspire to liberal whiteness." An article in The Yale Herald described it as a term "used to describe the ethnocentric politics of Asian Americans, usually of East Asian descent, who exclusively advocate for issues that benefit themselves, without acknowledging problematic dimensions of their own history and working to support other people of color." The feminist magazine Fem said that "the faces of boba liberalism are Asian Americans that are part of the middle and upper economic class. As a result, boba liberals disregard the negative effects of capitalism because they profit from it. For instance, boba liberals tend to focus on advocating for Asian representation in white spaces, or discussing whether or not wearing chopsticks in one's hair is culture appropriation. These topics are popular within boba liberal circles, all while dialogue regarding inequality, globalization, and racial injustice are purposely neglected." UnHerd notes that conservative Asian Americans have used the term not to critique capitalism, but to "aim at a small but influential group of progressive Asian-American activists who are supposedly selling out other Asians, especially working-class Asians, in order to win brownie points from elite, generally white liberals." MRAsians have similarly used the term to attack Asian American feminists who supported the Black Lives Matter movement. The Asian identity of boba liberals has often been accused of being shallow and superficial. Boba liberals are accused of using surface-level stereotypical Asian traits such as liking boba tea to bolster their Asian credentials. Plan A Magazine, an Asian diaspora magazine, described the film Crazy Rich Asians and the sitcom Fresh Off the Boat as "boba liberal media", calling them the result of "a specific kind of atomized identity politics". Other media outlets have connected the Crazy Rich Asians film to boba liberalism. == Controversy == The term "boba liberal" was coined in 2019 by Vietnamese American Twitter user Redmond (@diaspora_is_red) to analyze a form of Asian American liberalism through a Marxist lens. Redmond has criticized the misappropriation of their neologism by stripping away the Marxist framework by failing to discuss "socialism, communism, the capitalist system, imperialism, and the diaspora bourgeoisie" and conflating "boba liberalism" with the flawed concept of "East Asian privilege". In 2024, Redmond criticized misuse of the term by conservatives and liberals, and said "The term boba liberalism can go away for all I care. It's corny and stale". === United States === One commentator described boba liberals as supporting policies that primarily benefit upper-income Asian-Americans, and not necessarily the Asian-American community as a whole. Therefore, while the word "liberal" is used in the term, it is not mutually exclusive to one specific ideology, as it may also extend to conservative-aligned Asians in some areas, as they would often take advantage of the "model minority" label by defending such measures.
International Teletraffic Congress
The International Teletraffic Congress (ITC) is the first international conference in networking science and practice. It was created in 1955 by Arne Jensen to initially cater to the emerging need to understand and model traffic in telephone networks using stochastic methodologies, and to bring together researchers with these considerations as a common theme. Up through World War II, teletraffic research was done mainly by engineers and mathematicians working in telephone companies. Most of their work was published in local or company journals. In 1955, however, the field acquired a formal, international, institutional structure, with the organization of the first International Teletraffic Congress (ITC). Over the years, it has broaden its scope to address a wide spectrum ranging from the mathematical theory of traffic processes, stochastic system modelling and analysis, traffic and performance measurements, network management, traffic engineering to network capacity planning and cost optimization, including network economics and reliability for various types of networks. ITC served as a forum for all theoretical fundamentals and engineering practices for large-scale deployment and operation of telecommunications networks. Since its inception, ITC witnessed the evolution of communications and networking: the influence of computer science on telecommunication, the advent of the Internet and the massive deployment of mobile communications and optics, the appearance of peer-to-peer networking and social networks, the ever increasing speed and flexibility of new communication technologies, networks, user devices, and applications, and the ever changing operation challenges arising from this development. ITC documented this evolution with contemporary measurement studies, performance analyses of new technologies, recommendations for provisioning and configuration, and greatly contributed to the methodological toolbox of network scientists. Today, with its conferences, specialist seminars, regional seminars, training courses and publications, the ITC aims at a worldwide forum for all questions related to network and service performance, management, and assessment, both present and futuristic. The notion of traffic is broadly used to encompass data traffic from the MAC layer all the way to application traffic in the application layer. The scope of ITC is thus ranging all issues embedding operations, design, planning, economics and performance analysis of current and emerging communication networks and services, to be addressed by applying a variety of tools from different fields, such as Stochastic Processes, Information theory, Control theory, Signal and Processing, Game theory and optimization techniques, Statistical methodologies and Artificial Intelligence techniques. The target audience of such issues is experts from research organizations, universities, equipment vendors and suppliers, network operators, service providers, system integrators and international technical organizations, guaranteeing a well-balanced contribution from theory, application, and practice. The general goal remains to bring researchers and practitioners together toward operational understanding of all types of current and future networks. The ITC is ruled by the International Advisory Council (IAC) which gathers a number of technical experts, from universities and the research arms of key corporations in the industry, from countries having a strong tradition in teletraffic development. The IAC responsibilities are to disseminate information on teletraffic which is of interest for the whole community and: to select the locations of Plenary Congresses and to ensure their high-level technical programme to support Specialist Seminars on specific topics of current interest to promote Regional Seminars for the dissemination of teletraffic concepts in developing countries to facilitate the liaison activity with the ITU through participation in the standardization process and in the Development Programme The technical program and the organization of each ITC event remains within the responsibilities of the hosting country, but with significant IAC support to guarantee that the event is consistent with the quality standards established during the previous congresses. The ITC Plenary Congresses were scheduled tri-annually from 1955 until 1995 when the interval became bi-annual to account for the ever-accelerating development of network technologies, products and services and the associated dramatic increases in network demands. Similarly, to better cover the impact of dramatic changes undergoing in the field of computer and communication systems, networks and usage, it has been decided to hold the Plenary Congress on an annual basis from 2009. == Content == Teletraffic science is the traditional term for all theoretical fundamentals and engineering practices to describe data flows in telecommunication networks, the performance of the usage of network resources, procedures for sizing of resources and engineering the networks for given traffic load and quality of service requirements. For more than 50 years of the 20th century, traffic or teletraffic has been identified primarily with telephone networks. With the huge development of computers, stored program control of network nodes and computer communication, the traditional teletraffic science field naturally extended to computer networks, mobile and wireless/optical networks, and for a wide spectrum of new applications. The convergence between the voice network, the Internet, the television and mobility raised new questions that request new models and tools to be developed. In addition, the development of community networks, home networking, multiple access networking technologies, and the advent of pervasive and ambient communications dictates new challenges to be addressed. Today, ITC addresses the emerging paradigms such as an increasing diversity of distributed applications and services over various media like mobile/optical networks, enabling new markets and economy. ITC has steered the evolutions in communications since its creation in 1955 and remains at the forefront of innovation regarding modeling and performance. The scientific roots of communications traffic are based on the theory of probability and stochastic processes, modelling and performance evaluation. Modelling is the key for the mathematical description and quantitative performance analysis. Traffic flows are described by stochastic processes with complex dependencies which have to be validated by traffic measurements. Modelling also includes operational properties of resource control reflected by service strategies such as queueing disciplines, admission control, and routing. The results of such performance analyses are used for resource dimensioning (sizing), resource management, and network optimization while providing targeted Quality of Service. Teletraffic science is closely related to methods of operation research (queueing theory, optimization, forecasting) and computational sciences (simulation technology distributed systems). In this context, ITC represents a wide community of researchers and practitioners and is regularly organizing events like Congresses, Specialist Seminars and Workshops in order to discuss the latest changes in the modelling, design and performance of communication systems, networks and services. === The evolution of technologies of the 20th century === ITC has been witnessing the change of communication and networking technologies which are reflected in the proceedings and programs of the congresses. The specialist seminars and the motto of the congresses thereby reflect the hot topics of that time and the evolution. Selected topics of the 70's, 80's and 90's were 1998: Traffic Issues related to Multimedia and Nomadic Communications 1995: Traffic Modeling and Measurement in Broadband and Mobile Communications 1990: Broadband Technologies: Architectures, Applications, Control and Performance 1986: ISDN Traffic Issues 1984: Fundamentals of Teletraffic Theory 1977: Modeling of SPC Exchanges and Data Networks === Recent topics in the 21st century === With the rise of the Internet, new networking paradigms and technologies but also new challenges emerged: 2020: Teletraffic in the era of beyond-5G and AI 2019: Networked Systems and Services 2018: Teletraffic in the Smart World 2017: Ubiquitous, software-based, and sustainable networks and services 2016: Digital Connected World 2015: Traffic, Performance and Big Data 2014: Towards a Sustainable World 2013: Energy Efficient and Green Networking 2010: Multimedia Applications - Traffic, Performance and QoE 2009: Network Virtualization - Concepts and Performance 2008: Future Internet Design and Experimental Facilities 2008: Quality of Experience 2002: Internet Traffic Engineering and Traffic Management == Arne Jensen Lifetime Achievement Awards == The Arne Jensen Lifetime A
Librem
Librem is a line of computers manufactured by Purism, SPC featuring free (libre) software. The laptop line is designed to protect privacy and freedom by omitting non-free (proprietary) software in their operating system and kernel, avoiding the Intel Active Management Technology, and gradually freeing and securing firmware. Librem laptops feature hardware kill switches for the microphone, webcam, Bluetooth and Wi-Fi. == Models == === Laptops === ==== Librem 13, Librem 15 and Librem 14 ==== In 2014, Purism launched a crowdfunding campaign on Crowd Supply to fund the creation and production of the Librem 15 laptop, conceived as a modern alternative to existing open-source hardware laptops, all of which used older hardware. The 15 in the name refers to its 15-inch screen size. The campaign succeeded after extending the original campaign, and the laptops were shipped to backers. In a second revision of the laptop, hardware kill switches for the camera, microphone, Wi-Fi, and Bluetooth were added. After the successful launch of the Librem 15, Purism created another campaign on Crowd Supply for a 13-inch laptop named Librem 13, which also came with hardware kill switches similar to those on the Librem 15v2. The campaign was again successful and the laptops were shipped to customers. Purism announced in December 2016 that it would start shipping from inventory rather than building to order with the new batches of Librem 15 and 13. As of January 2023, Purism has one laptop model in production, the Librem 14. ==== Comparison of laptops ==== === Librem Mini === The Librem Mini is a small form factor desktop computer, which began shipping in June 2020. === Librem 5 === On August 24, 2017, Purism began a crowdfunding campaign for the Librem 5, a smartphone aimed to run 100% free software, which would "[focus] on security by design and privacy protection by default". Purism claimed that the phone would become "the world's first ever IP-native mobile handset, using end-to-end encrypted decentralized communication." Purism cooperated with KDE and GNOME in its development of Librem 5. Security features of the Librem 5 include separation of the CPU from the baseband processor, which, according to Linux Magazine, makes the Librem 5 unique in comparison to other mobile phones. The Librem 5 also features hardware kill switches for Wi-Fi and Bluetooth communication and the phone's camera, microphone, and baseband processor. The default operating system for the Librem 5 is Purism's PureOS, a Debian derivative. The operating system uses a new user interface named Phosh, based on Wayland, wlroots, GTK and GNOME middleware. It is planned that Phosh/Plasma Mobile, Ubuntu Touch, and postmarketOS can also be installed on the phone. The release of the Librem 5 has been postponed several times. In September 2018, Purism announced that the launch date of Librem 5 would be moved from January to April 2019, because of two hardware bugs and the holiday season in Europe and North America. The Librem 5's DevKits for software developers were shipped in December 2018. The launch date was later postponed to the third quarter because of the necessity of further CPU tests. On September 24, 2019, Purism announced that the first batch of Librem 5 phones had begun shipping. The finished version of the Librem 5, known as "Evergreen", was finally shipped on November 18, 2020. === Librem Server === The Librem server is a rack mounted server, released to the public in December 2019. === Librem Key === Announced on 20 September 2018, the Librem Key is a hardware USB security token with multiple features, including integration with a tamper-evident Heads BIOS, which ensures that the Librem laptop Basic Input/Output System (BIOS) was not maliciously altered since the last laptop launch. The Librem Key also features one-time password storage with 3x HMAC-based One-time Password algorithm (HOTP) (RFC 4226) and 15 x Time-based One-time Password algorithm (TOTP) (RFC 6238) and an integrated password manager (16 entries), 40 kbit/s true random number generator, and a tamper-resistant smart card. The key supports type A USB 2.0, has dimensions of 48 x 19 x 7 mm, and weighs 6 g. == Operating system == Initially planning to preload its Librem laptops with the Trisquel operating system, Purism eventually moved off the Trisquel platform to Debian for the 2.0 release of its PureOS Linux operating system. As an alternative to PureOS, Librem laptops are purchasable with Qubes OS preinstalled. In December 2017, the Free Software Foundation added PureOS to its list of endorsed GNU/Linux distributions. == BIOS == In 2015, Purism began research to port the Librem 13 to coreboot but the effort was initially stalled. By the end of the year, a coreboot developer completed an initial port of the Librem 13 and submitted it for review. In December 2016, hardware enablement developer Youness Alaoui joined Purism and was tasked to complete the coreboot port for the original Librem 13 and prepare a port for the second revision of the device. Since summer 2017, new Librem laptops are shipped with coreboot as their standard BIOS, and updates are available for all older models. Purism calls a collection of these six components, involved in the boot process, as PureBoot: Neutralized and disabled Intel Management Engine coreboot A Trusted Platform Module (TPM) chip Heads, which has tamper-evident features to detect if the BIOS or important boot files have been modified Librem Key, Purism's USB security token Multi-factor authentication that unlocks disk encryption using the Librem Key PureBoot protects the users from various attacks like theft, BIOS malware and kernel rootkits, vulnerabilities and malicious code in the Intel Management Engine, and interdiction.
Tea (app)
Tea, officially Tea Dating Advice, is a dating surveillance mobile phone application that allows women to post personal data about men they are interested in or are currently dating. Founded by Sean Cook, the app rose to prominence in July 2025 after it was the subject of three major data leaks in July and August 2025. It was removed from Apple's App Store in October 2025, but remains available on the Google Play Store. == History == The app enables its users to upload, view, and comment on photos of men, check men's public records, and perform image searches. It also provides the ability to rate and review men, as well as a group chat function. The app uses artificial intelligence to verify that the user is a woman through facial analysis and other personal information to preserve the app as a women-only space. Users are required to submit their photo and an ID to access the app. The company that created the app was founded by businessman and tech capitalist Sean Cook, who stated in July 2025 that he was inspired to create the app because of his mother's experiences from online dating. According to the company, users remain anonymous, and the requirement to upload an ID was removed in 2023. An August 2025 investigation by 404 Media suggested that much of the information given by Cook on the historical background of the company was inaccurate. In July 2025, private messages, other personally identifying information, and approximately 72,000 images were leaked via 4chan. A further 1.1 million private messages were subsequently leaked using a separate security vulnerability; these included intimate conversations about controversial topics such as adultery and other forms of infidelity to their partners, discussions of abortion, phone numbers, meeting locations, and other confidential communications. The app's publishers subsequently revoked the ability to private message users in the app. Shortly after, the app was hidden from search on Android and an interactive, unverified map was also created of those in the files. By 7 August 2025, ten class action lawsuits had been filed. A further leak was reported later that month. Proponents have praised the app as an aid for women's safety by helping them check men for adultery, catfishing, criminal convictions and other "red flag" behaviors. Critics have described the app as a doxing tool and a violation of privacy, an opportunity for defamation against innocent individuals, and a witch hunt. Cook has stated that the company's legal team receives about three legal threats per day. Another mobile app, called TeaOnHer, was created in response of the app’s popularity. It was described as the male version of the Tea app. The app also reported a data breach in August 2025. In October 2025, Apple removed the app from their app store, telling journalists that the removal was due to a failure to meet company terms regarding content moderation and user privacy. Apple also mentioned an excessive amount of complaints, including allegations that the personal information of minors was being shared. The app remains on the Google Play Store.
Randomized benchmarking
Randomized benchmarking is an experimental method for measuring the average error rates of quantum computing hardware platforms. The protocol estimates the average error rates by implementing long sequences of randomly sampled quantum gate operations. Randomized benchmarking is the industry-standard protocol used by quantum hardware developers such as IBM and Google to test the performance of the quantum operations. The original theory of randomized benchmarking, proposed by Joseph Emerson and collaborators, considered the implementation of sequences of Haar-random operations, but this had several practical limitations. The now-standard protocol for randomized benchmarking (RB) relies on uniformly random Clifford operations, as proposed in 2006 by Dankert et al. as an application of the theory of unitary t-designs. In current usage randomized benchmarking sometimes refers to the broader family of generalizations of the 2005 protocol involving different random gate sets that can identify various features of the strength and type of errors affecting the elementary quantum gate operations. Randomized benchmarking protocols are an important means of verifying and validating quantum operations and are also routinely used for the optimization of quantum control procedures. == Overview == Randomized benchmarking offers several key advantages over alternative approaches to error characterization. For example, the number of experimental procedures required for full characterization of errors (called tomography) grows exponentially with the number of quantum bits (called qubits). This makes tomographic methods impractical for even small systems of just 3 or 4 qubits. In contrast, randomized benchmarking protocols are the only known approaches to error characterization that scale efficiently as number of qubits in the system increases. Thus RB can be applied in practice to characterize errors in arbitrarily large quantum processors. Additionally, in experimental quantum computing, procedures for state preparation and measurement (SPAM) are also error-prone, and thus quantum process tomography is unable to distinguish errors associated with gate operations from errors associated with SPAM. In contrast, RB protocols are robust to state-preparation and measurement errors Randomized benchmarking protocols estimate key features of the errors that affect a set of quantum operations by examining how the observed fidelity of the final quantum state decreases as the length of the random sequence increases. If the set of operations satisfies certain mathematical properties, such as comprising a sequence of twirls with unitary two-designs, then the measured decay can be shown to be an invariant exponential with a rate fixed uniquely by features of the error model. == History == Randomized benchmarking was proposed in Scalable noise estimation with random unitary operators, where it was shown that long sequences of quantum gates sampled uniformly at random from the Haar measure on the group SU(d) would lead to an exponential decay at a rate that was uniquely fixed by the error model. Emerson, Alicki and Zyczkowski also showed, under the assumption of gate-independent errors, that the measured decay rate is directly related to an important figure of merit, the average gate fidelity and independent of the choice of initial state and any errors in the initial state, as well as the specific random sequences of quantum gates. This protocol applied for arbitrary dimension d and an arbitrary number n of qubits, where d=2n. The SU(d) RB protocol had two important limitations that were overcome in a modified protocol proposed by Dankert et al., who proposed sampling the gate operations uniformly at random from any unitary two-design, such as the Clifford group. They proved that this would produce the same exponential decay rate as the random SU(d) version of the protocol proposed in Emerson et al.. This follows from the observation that a random sequence of gates is equivalent to an independent sequence of twirls under that group, as conjectured in and later proven in. This Clifford-group approach to Randomized Benchmarking is the now standard method for assessing error rates in quantum computers. A variation of this protocol was proposed by NIST in 2008 for the first experimental implementation of an RB-type for single qubit gates. However, the sampling of random gates in the NIST protocol was later proven not to reproduce any unitary two-design. The NIST RB protocol was later shown to also produce an exponential fidelity decay, albeit with a rate that depends on non-invariant features of the error model In recent years a rigorous theoretical framework has been developed for Clifford-group RB protocols to show that they work reliably under very broad experimental conditions. In 2011 and 2012, Magesan et al. proved that the exponential decay rate is fully robust to arbitrary state preparation and measurement errors (SPAM). They also proved a connection between the average gate fidelity and diamond norm metric of error that is relevant to the fault-tolerant threshold. They also provided evidence that the observed decay was exponential and related to the average gate fidelity even if the error model varied across the gate operations, so-called gate-dependent errors, which is the experimentally realistic situation. In 2018, Wallman and Dugas et al., showed that, despite concerns raised in, even under very strong gate-dependence errors the standard RB protocols produces an exponential decay at a rate that precisely measures the average gate-fidelity of the experimentally relevant errors. The results of Wallman. in particular proved that the RB error rate is so robust to gate-dependent errors models that it provides an extremely sensitive tool for detecting non-Markovian errors. This follows because under a standard RB experiment only non-Markovian errors (including time-dependent Markovian errors) can produce a statistically significant deviation from an exponential decay The standard RB protocol was first implemented for single qubit gate operations in 2012 at Yale on a superconducting qubit. A variation of this standard protocol that is only defined for single qubit operations was implemented by NIST in 2008 on a trapped ion. The first implementation of the standard RB protocol for two-qubit gates was performed in 2012 at NIST for a system of two trapped ions