Data profiling is the process of examining the data available from an existing information source (e.g. a database or a file) and collecting statistics or informative summaries about that data. The purpose of these statistics may be to: Find out whether existing data can be easily used for other purposes Improve the ability to search data by tagging it with keywords, descriptions, or assigning it to a category Assess data quality, including whether the data conforms to particular standards or patterns Assess the risk involved in integrating data in new applications, including the challenges of joins Discover metadata of the source database, including value patterns and distributions, key candidates, foreign-key candidates, and functional dependencies Assess whether known metadata accurately describes the actual values in the source database Understanding data challenges early in any data intensive project, so that late project surprises are avoided. Finding data problems late in the project can lead to delays and cost overruns. Have an enterprise view of all data, for uses such as master data management, where key data is needed, or data governance for improving data quality. == Introduction == Data profiling refers to the analysis of information for use in a data warehouse in order to clarify the structure, content, relationships, and derivation rules of the data. Profiling helps to not only understand anomalies and assess data quality, but also to discover, register, and assess enterprise metadata. The result of the analysis is used to determine the suitability of the candidate source systems, usually giving the basis for an early go/no-go decision, and also to identify problems for later solution design. == How data profiling is conducted == Data profiling utilizes methods of descriptive statistics such as minimum, maximum, mean, mode, percentile, standard deviation, frequency, variation, aggregates such as count and sum, and additional metadata information obtained during data profiling such as data type, length, discrete values, uniqueness, occurrence of null values, typical string patterns, and abstract type recognition. The metadata can then be used to discover problems such as illegal values, misspellings, missing values, varying value representation, and duplicates. Different analyses are performed for different structural levels. E.g. single columns could be profiled individually to get an understanding of frequency distribution of different values, type, and use of each column. Embedded value dependencies can be exposed in a cross-columns analysis. Finally, overlapping value sets possibly representing foreign key relationships between entities can be explored in an inter-table analysis. Normally, purpose-built tools are used for data profiling to ease the process. The computational complexity increases when going from single column, to single table, to cross-table structural profiling. Therefore, performance is an evaluation criterion for profiling tools. == When is data profiling conducted? == According to Kimball, data profiling is performed several times and with varying intensity throughout the data warehouse developing process. A light profiling assessment should be undertaken immediately after candidate source systems have been identified and DW/BI business requirements have been satisfied. The purpose of this initial analysis is to clarify at an early stage if the correct data is available at the appropriate detail level and that anomalies can be handled subsequently. If this is not the case the project may be terminated. Additionally, more in-depth profiling is done prior to the dimensional modeling process in order assess what is required to convert data into a dimensional model. Detailed profiling extends into the ETL system design process in order to determine the appropriate data to extract and which filters to apply to the data set. Additionally, data profiling may be conducted in the data warehouse development process after data has been loaded into staging, the data marts, etc. Conducting data at these stages helps ensure that data cleaning and transformations have been done correctly and in compliance of requirements. == Benefits and examples == Data profiling can improve data quality, shorten the implementation cycle of major projects, and improve users' understanding of data. Discovering business knowledge embedded in data itself is one of the significant benefits derived from data profiling. It can improve data accuracy in corporate databases.
Attensity
Attensity was an American company that provided social analytics and engagement applications for social customer relationship management (social CRM). Attensity's text analytics software applications extracted facts, relationships and sentiment from unstructured data. == History == Attensity was founded in 2000. An early investor in Attensity was In-Q-Tel, which funds technology to support the missions of the US Government and the broader DOD. InTTENSITY, an independent company that has combined Inxight with Attensity Software (the only joint development project that combines two InQTel funded software packages), was the exclusive distributor and outlet for Attensity in the Federal Market. In 2009, Attensity Corp., then based in Palo Alto, merged with Germany's Empolis and Living-e AG to form Attensity Group. In 2010, Attensity Group acquired Biz360, a provider of social media monitoring and market intelligence solutions. In early 2012, Attensity Group divested itself of the Empolis business unit via a management buyout; that unit currently conducts business under its pre-merger name. Attensity Group was a closely held private company. Its majority shareholder was Aeris Capital, a private Swiss investment office advising a high-net-worth individual and his charitable foundation. Foundation Capital, Granite Ventures, and Scale Venture Partners were among Biz360's investors and thus became shareholders in Attensity Group. In February 2016, Attensity's IP assets were acquired by InContact, and Attensity closed.
Artificial intelligence in India
The artificial intelligence (AI) market in India is projected to reach $8 billion by 2025, growing at 40% CAGR from 2020 to 2025. This growth is part of the broader AI boom, a global period of rapid technological advancements with India being pioneer starting in the early 2010s with NLP based Chatbots from Haptik, Corover.ai, Niki.ai and then gaining prominence in the early 2020s based on reinforcement learning, marked by breakthroughs such as generative AI models from Krutrim, Sarvam, CoRover, OpenAI and Alphafold by Google DeepMind. In India, the development of AI has been similarly transformative, with applications in healthcare, finance, and education, bolstered by government initiatives like NITI Aayog's 2018 National Strategy for Artificial Intelligence. Institutions such as the Indian Statistical Institute and the Indian Institute of Science published breakthrough AI research papers and patents. India's transformation to AI is primarily being driven by startups and government initiatives & policies like Digital India. By fostering technological trust through digital public infrastructure, India is tackling socioeconomic issues by taking a bottom-up approach to AI. NASSCOM and Boston Consulting Group estimate that by 2027, India's AI services might be valued at $17 billion. According to 2025 Technology and Innovation Report, by UN Trade and Development, India ranks 10th globally for private sector investments in AI. According to Mary Meeker, India has emerged as a key market for AI platforms, accounting for the largest share of ChatGPT's mobile app users and having the third-largest user base for DeepSeek in 2025. While AI presents significant opportunities for economic growth and social development in India, challenges such as data privacy concerns, skill shortages, and ethical considerations need to be addressed for responsible AI deployment. The growth of AI in India has also led to an increase in the number of cyberattacks that use AI to target organizations. == History == === Early days (1960s-1980s) === The TIFRAC (Tata Institute of Fundamental Research Automatic Calculator) was designed and developed by a team led by Rangaswamy Narasimhan between 1954 and 1960. He worked on pattern recognition from 1961 to 1964 at the University of Illinois Urbana-Champaign's Digital Computer Laboratory. In order to conduct research on database technology, computer networking, computer graphics, and systems software, he and M. G. K. Menon founded the National Centre for Software Development and Computing Techniques. In 1965, he established the Computer Society of India and supervised the initial research work on AI at Tata Institute of Fundamental Research. Jagdish Lal launched the first computer science program in 1976 at Motilal Nehru Regional Engineering College. H. K. Kesavan from the University of Waterloo and Vaidyeswaran Rajaraman from the University of Wisconsin–Madison joined the IIT Kanpur Electrical Engineering Department in 1963–1964 as Assistant Professor and Head of Department, respectively. H.N. Mahabala, who was employed at Bendix Corporation's Computer Division, joined the department in 1965. He previously worked with Marvin Minsky. The IIT Kanpur Computer Center was led by H. K. Kesavan, with Vaidyeswaran Rajaraman serving as his deputy. Kesavan informally permitted Rajaraman and Mahabala to introduce artificial intelligence into computer science classes. The computer science program was approved by IIT Kanpur in 1971 and split out from the electrical engineering department. In 1973, an IBM System/370 Model 155 was installed at IIT Madras. John McCarthy, head of the Artificial Intelligence Laboratory at Stanford University visited IIT Kanpur in 1971. He donated PDP-1 with a time-sharing operating system. During the 1970s, the balance of payments deficit in India restricted import of computers. The Department of Computer Science and Automation at the Indian Institute of Science established in 1969, played an important role in nurturing the development of data science and artificial intelligence in India. First course on AI was introduced in the 1970s by G. Krishna. B. L. Deekshatulu introduced the first course on pattern recognition in the early 1970s. === Foundation phase === ==== 1980s ==== In the 1980s, the Indian Statistical Institute's Optical Character Recognition Project was one of the country's first attempts at studying artificial intelligence and machine learning. OCR technology has benefited greatly from the work of ISI's Computer Vision and Pattern Recognition Unit, which is headed by Bidyut Baran Chaudhuri. He also contributed in the development of computer vision and digital image processing. As part of the Indian Fifth Generation Computer Systems Research Programme, the Department of Electronics, with support from the United Nations Development Programme, initiated the Knowledge Based Computer Systems Project in 1986, marking the beginning of India's first major AI research program. Prime Minister Rajiv Gandhi requested that the Department of Electronics and IISc to initiate the Parallel Processing Project in 1986–1987. The Center for Development of Advanced Computing eventually joined those efforts. IIT Madras was selected to develop system diagnosis, ISI for image processing, National Centre for Software Technology for natural language processing and TIFR for speech processing. In 1987, the proposal of N. Seshagiri, Director General of the National Informatics Centre for the prototype development of supercomputer was cleared. Negotiations for a Cray supercomputer were underway between the Reagan administration and the Rajiv Gandhi government. US Defense Secretaries Frank Carlucci and Caspar Weinberger visited New Delhi after the US approved the transfer in 1988. The sale of a lower-end XMP-14 supercomputer was permitted in lieu of the Cray XMP-24 supercomputer due to security concerns. The Center for Development of Advanced Computing was formally established in March 1988 by the Ministry of Communications and Information Technology (previously the Ministry of IT) within the Department of Information Technology (formerly the Department of Electronics) in response to a recommendation made to the Prime Minister by the Scientific Advisory Council. The National Initiative in Supercomputing, which produced the PARAM series, was led by Vijay P. Bhatkar. For the first ten years, supercomputing and Indian language computing were the two main focus areas. C-DAC has expanded its operations in order to meet the needs in a number of domains, including network and internet software, real-time systems, artificial intelligence, and NLP. Under the direction of Professor KV Ramakrishnamacharyulu from National Sanskrit University and Professor Rajeev Sangal from the International Institute of Information Technology, Hyderabad, the Akshar Bharati Research Group was established in 1984 with support from IIT Kanpur and the University of Hyderabad for computational processing of Indian languages. They focused on computational linguistics, NLP with ontological database systems, and Indian language/translation theories with linguistic tradition. ==== 1990s ==== From IIT Kanpur, Mohan Tambe joined C-DAC in the 1990s to work on Graphics and Intelligence based Script Technology (GIST), which addressed the challenge of adapting personal computer software based on Latin script to Devanagiri and a number of other Indian language scripts. He was previously working on the Machine Translation for Indian languages Project. Within C-DAC, he established the GIST group. The technology was expanded to encompass NLP, artificial intelligence-based machine-aided language learning and translation, multimedia and multilingual computing solutions, and more. GIST resulted in the creation of G-CLASS (GIST cross language search plug-ins suite), a cross-language search engine. The Applied Artificial Intelligence Group at C-DAC has developed some basic and novel applications in the field of NLP, including machine translation, information extraction/retrieval, automatic summarization, speech recognition, text-to-speech synthesis, intelligent language teaching, and natural language-based document management with Decision Support Systems. These applications are the result of the foundation laid by previous language technology activities. Software firms in the Indian private sector began looking into AI applications, mostly in the area of business process automation. In order to allow machines to read, comprehend, and interpret human languages, the Language Technologies Research Center was founded in October 1999 at the International Institute of Information Technology, Hyderabad. It focused on the advancements in semantic parsing, information extraction, natural language generation, sentiment analysis, and dialogue systems. Some of the early AI research in India was driven by societal needs. For example; Eklavya, a knowledge-based program created by I
Timeline of algorithms
The following timeline of algorithms outlines the development of algorithms (mainly "mathematical recipes") since their inception. == Antiquity == Before – writing about "recipes" (on cooking, rituals, agriculture and other themes) c. 1700–2000 BC – Egyptians develop earliest known algorithms for multiplying two numbers c. 1600 BC – Babylonians develop earliest known algorithms for factorization and finding square roots c. 300 BC – Euclid's algorithm c. 200 BC – the Sieve of Eratosthenes 263 AD – Gaussian elimination described by Liu Hui == Medieval Period == 628 – Chakravala method described by Brahmagupta c. 820 – Al-Khawarizmi described algorithms for solving linear equations and quadratic equations in his Algebra; the word algorithm comes from his name 825 – Al-Khawarizmi described the algorism, algorithms for using the Hindu–Arabic numeral system, in his treatise On the Calculation with Hindu Numerals, which was translated into Latin as Algoritmi de numero Indorum, where "Algoritmi", the translator's rendition of the author's name gave rise to the word algorithm (Latin algorithmus) with a meaning "calculation method" c. 850 – cryptanalysis and frequency analysis algorithms developed by Al-Kindi (Alkindus) in A Manuscript on Deciphering Cryptographic Messages, which contains algorithms on breaking encryptions and ciphers c. 1025 – Ibn al-Haytham (Alhazen), was the first mathematician to derive the formula for the sum of the fourth powers, and in turn, he develops an algorithm for determining the general formula for the sum of any integral powers c. 1400 – Ahmad al-Qalqashandi gives a list of ciphers in his Subh al-a'sha which include both substitution and transposition, and for the first time, a cipher with multiple substitutions for each plaintext letter; he also gives an exposition on and worked example of cryptanalysis, including the use of tables of letter frequencies and sets of letters which can not occur together in one word == Before 1940 == 1540 – Lodovico Ferrari discovered a method to find the roots of a quartic polynomial 1545 – Gerolamo Cardano published Cardano's method for finding the roots of a cubic polynomial 1614 – John Napier develops method for performing calculations using logarithms 1671 – Newton–Raphson method developed by Isaac Newton 1690 – Newton–Raphson method independently developed by Joseph Raphson 1706 – John Machin develops a quickly converging inverse-tangent series for π and computes π to 100 decimal places 1768 – Leonhard Euler publishes his method for numerical integration of ordinary differential equations in problem 85 of Institutiones calculi integralis 1789 – Jurij Vega improves Machin's formula and computes π to 140 decimal places, 1805 – FFT-like algorithm known by Carl Friedrich Gauss 1842 – Ada Lovelace writes the first algorithm for a computing engine 1903 – A fast Fourier transform algorithm presented by Carle David Tolmé Runge 1918 - Soundex 1926 – Borůvka's algorithm 1926 – Primary decomposition algorithm presented by Grete Hermann 1927 – Hartree–Fock method developed for simulating a quantum many-body system in a stationary state. 1934 – Delaunay triangulation developed by Boris Delaunay 1936 – Turing machine, an abstract machine developed by Alan Turing, with others developed the modern notion of algorithm. == 1940s == 1942 – A fast Fourier transform algorithm developed by G.C. Danielson and Cornelius Lanczos 1945 – Merge sort developed by John von Neumann 1947 – Simplex algorithm developed by George Dantzig == 1950s == 1950 – Hamming codes developed by Richard Hamming 1952 – Huffman coding developed by David A. Huffman 1953 – Simulated annealing introduced by Nicholas Metropolis 1954 – Radix sort computer algorithm developed by Harold H. Seward 1964 – Box–Muller transform for fast generation of normally distributed numbers published by George Edward Pelham Box and Mervin Edgar Muller. Independently pre-discovered by Raymond E. A. C. Paley and Norbert Wiener in 1934. 1956 – Kruskal's algorithm developed by Joseph Kruskal 1956 – Ford–Fulkerson algorithm developed and published by R. Ford Jr. and D. R. Fulkerson 1957 – Prim's algorithm developed by Robert Prim 1957 – Bellman–Ford algorithm developed by Richard E. Bellman and L. R. Ford, Jr. 1959 – Dijkstra's algorithm developed by Edsger Dijkstra 1959 – Shell sort developed by Donald L. Shell 1959 – De Casteljau's algorithm developed by Paul de Casteljau 1959 – QR factorization algorithm developed independently by John G.F. Francis and Vera Kublanovskaya 1959 – Rabin–Scott powerset construction for converting NFA into DFA published by Michael O. Rabin and Dana Scott == 1960s == 1960 – Karatsuba multiplication 1961 – CRC (Cyclic redundancy check) invented by W. Wesley Peterson 1962 – AVL trees 1962 – Quicksort developed by C. A. R. Hoare 1962 – Bresenham's line algorithm developed by Jack E. Bresenham 1962 – Gale–Shapley 'stable-marriage' algorithm developed by David Gale and Lloyd Shapley 1964 – Heapsort developed by J. W. J. Williams 1964 – multigrid methods first proposed by R. P. Fedorenko 1965 – Cooley–Tukey algorithm rediscovered by James Cooley and John Tukey 1965 – Levenshtein distance developed by Vladimir Levenshtein 1965 – Cocke–Younger–Kasami (CYK) algorithm independently developed by Tadao Kasami 1965 – Buchberger's algorithm for computing Gröbner bases developed by Bruno Buchberger 1965 – LR parsers invented by Donald Knuth 1966 – Dantzig algorithm for shortest path in a graph with negative edges 1967 – Viterbi algorithm proposed by Andrew Viterbi 1967 – Cocke–Younger–Kasami (CYK) algorithm independently developed by Daniel H. Younger 1968 – A graph search algorithm described by Peter Hart, Nils Nilsson, and Bertram Raphael 1968 – Risch algorithm for indefinite integration developed by Robert Henry Risch 1969 – Strassen algorithm for matrix multiplication developed by Volker Strassen == 1970s == 1970 – Dinic's algorithm for computing maximum flow in a flow network by Yefim (Chaim) A. Dinitz 1970 – Knuth–Bendix completion algorithm developed by Donald Knuth and Peter B. Bendix 1970 – BFGS method of the quasi-Newton class 1970 – Needleman–Wunsch algorithm published by Saul B. Needleman and Christian D. Wunsch 1972 – Edmonds–Karp algorithm published by Jack Edmonds and Richard Karp, essentially identical to Dinic's algorithm from 1970 1972 – Graham scan developed by Ronald Graham 1972 – Red–black trees and B-trees discovered 1973 – RSA encryption algorithm discovered by Clifford Cocks 1973 – Jarvis march algorithm developed by R. A. Jarvis 1973 – Hopcroft–Karp algorithm developed by John Hopcroft and Richard Karp 1974 – Pollard's p − 1 algorithm developed by John Pollard 1974 – Quadtree developed by Raphael Finkel and J.L. Bentley 1975 – Genetic algorithms popularized by John Holland 1975 – Pollard's rho algorithm developed by John Pollard 1975 – Aho–Corasick string matching algorithm developed by Alfred V. Aho and Margaret J. Corasick 1975 – Cylindrical algebraic decomposition developed by George E. Collins 1976 – Salamin–Brent algorithm independently discovered by Eugene Salamin and Richard Brent 1976 – Knuth–Morris–Pratt algorithm developed by Donald Knuth and Vaughan Pratt and independently by J. H. Morris 1977 – Boyer–Moore string-search algorithm for searching the occurrence of a string into another string. 1977 – RSA encryption algorithm rediscovered by Ron Rivest, Adi Shamir, and Len Adleman 1977 – LZ77 algorithm developed by Abraham Lempel and Jacob Ziv 1977 – multigrid methods developed independently by Achi Brandt and Wolfgang Hackbusch 1978 – LZ78 algorithm developed from LZ77 by Abraham Lempel and Jacob Ziv 1978 – Bruun's algorithm proposed for powers of two by Georg Bruun 1979 – Khachiyan's ellipsoid method developed by Leonid Khachiyan 1979 – ID3 decision tree algorithm developed by Ross Quinlan == 1980s == 1980 – Brent's Algorithm for cycle detection Richard P. Brendt 1981 – Quadratic sieve developed by Carl Pomerance 1981 – Smith–Waterman algorithm developed by Temple F. Smith and Michael S. Waterman 1983 – Simulated annealing developed by S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi 1983 – Classification and regression tree (CART) algorithm developed by Leo Breiman, et al. 1984 – LZW algorithm developed from LZ78 by Terry Welch 1984 – Karmarkar's interior-point algorithm developed by Narendra Karmarkar 1984 – ACORN PRNG discovered by Roy Wikramaratna and used privately 1985 – Simulated annealing independently developed by V. Cerny 1985 – Car–Parrinello molecular dynamics developed by Roberto Car and Michele Parrinello 1985 – Splay trees discovered by Sleator and Tarjan 1986 – Blum Blum Shub proposed by L. Blum, M. Blum, and M. Shub 1986 – Push relabel maximum flow algorithm by Andrew Goldberg and Robert Tarjan 1986 – Barnes–Hut tree method developed by Josh Barnes and Piet Hut for fast approximate simulation of n-body problems 1987 – Fast multipole method developed by Leslie Greengard and Vladimir
Data Science Africa
Data Science Africa (DSA) is a non-profit knowledge sharing professional group that aims at bringing together leading researchers and practitioners working on data science methods or applications relevant to Africa, and providing training on state of the art data science methods to students and others interested in developing practical skills. Since 2013, DSA has been organizing conference, workshops and summer schools on machine learning and data science across East Africa. Facilitators of Summer School and workshops are researchers and practitioners from the academia, private and public institutions across the world. == Summer schools and workshops == The first summer school which started as Gaussian Process Summer School was held at Makerere University in Kampala, Uganda from 6th to 9 August 2013. The First Data Science Summer School and Workshop was held at Dedan Kimathi University of Technology in Nyeri, Kenya from 15th to 19 June 2015. The Second Data Science Summer School was held at Makerere University, Kampala, Uganda from 27th to 29 July 2016, and the workshop was held at Pulse Lab, Kampala, Uganda from 30 July to 1 August 2016. The Third Data Science Summer School and Workshop was held at Nelson Mandela African Institute of Science and Technology, Tanzania from 19th to 21 July 2017. Among the sponsors of the event was ARM
Spatiotemporal reservoir resampling
Spatiotemporal reservoir resampling, commonly known as ReSTIR (from "Reservoir-based SpatioTemporal Importance Resampling"), is a collection of computer graphics techniques for reusing samples during rendering. It was developed primarily to allow more realistic lighting in real-time rendering, because relatively few rays can be traced per pixel while maintaining an acceptable frame rate. It can also be used to speed up off-line path tracing. The first ReSTIR paper, published in 2020, provided algorithms for direct lighting, allowing scenes containing thousands of lights to be rendered in real time on a high-end GPU. Researchers later proposed versions for rendering indirect lighting (and more recently, motion blur and depth of field) and built up a framework of mathematical concepts and notation conventions that help analyze such algorithms. A major focus of this work is removing or reducing the bias that could be introduced when samples from other pixels or frames are reused—or selectively allowing some bias in order to speed up rendering and reduce variance (visible as "noise" in the image). Versions for path tracing apply transformations called shift mappings to samples, typically reusing parts of paths closer to the light and modifying the portion closer to the camera. ReSTIR-related papers and talks have been presented every year at the SIGGRAPH conference since 2020. One of the first games to incorporate ReSTIR into its rendering was Cyberpunk 2077. == Overview and motivation == According to Chris Wyman, one of the co-authors of the original paper, although developers commonly thought that bias was acceptable for real-time rendering, end users (e.g. gamers) are well-aware of the artifacts caused by bias and many have a negative opinion of common sample-reuse techniques such as temporal anti-aliasing (TAA), which may cause "ghosting" when the camera moves, and denoising, which causes blurring and other artifacts. ReSTIR techniques can reduce or avoid these types of bias by reusing samples of the set of possible paths taken by light to reach the camera, instead of reusing rendered pixel color values (which are typically the average of multiple samples, discarding information such as the direction of the light). While other techniques reuse samples in a generic post-processing step, ReSTIR passes can test for shadowing, and reused samples are converted into pixel color values by rendering code that takes the characteristics of different materials into account (e.g. by implementing BRDFs). However the output of ReSTIR is noisy, and a denoising pass is typically still used. Stochastic ray tracing techniques such as path tracing need to average multiple samples (produced by tracing individual rays) in order to render a visually acceptable image. When using a simple unbiased renderer based on Monte Carlo integration, halving the deviation of the result (apparent as "noise" in the image) requires multiplying the number of samples by four, meaning that a rapidly increasingly number of samples is needed to improve quality, Standard ways to mitigate this problem include importance sampling (which requires finding improved sampling distributions for specific situations), and quasi-Monte Carlo integration (which usually still requires tracing a large number of rays). ReSTIR offers a solution that multiplies the effective number of samples while tracing a fixed number of additional rays per frame. Temporal reuse multiplies the effective sample count by the number of frames rendered. Spatial reuse multiplies the effective count by the number of neighboring pixels examined. These two types of reuse can be combined, allowing spatial reuse to be applied recursively, which appears to offer an exponentially increasing effective sample count, however this is quickly limited by the size of the neighborhood used for spatial reuse. Spatial reuse is also potentially less effective near shadow and object edges, especially for objects with fine geometric detail, and temporal reuse is limited by movement of the camera and scene elements. == Variations == Many variations of ReSTIR have been proposed that generalize or improve the original technique (which builds on an earlier method called RIS), specialize it for particular types of illumination or other visual effects, or allow incorporation into rendering algorithms other than standard path tracing. Some published versions are listed below. == Algorithms == === Basic algorithm === ReSTIR uses a combination of resampled importance sampling (RIS) and weighted reservoir sampling (WRS) which the authors call streaming RIS. RIS processes samples from an initial probability distribution (e.g. a probability distribution for which a cheap sampling method exists) and generates samples in a new probability distribution (e.g. a sampling distribution that is optimal for rendering but is impractical to draw samples from directly). WRS allows this to be done while storing only a small number of samples in memory, which is especially helpful on a GPU. Information about the samples is stored in a data structure called a reservoir. WRS also allows samples from multiple reservoirs to be combined ("merged") into a single reservoir; this is crucial for sample reuse. Each pixel has a reservoir, typically containing only a single sample when ReSTIR is used for real-time rendering (some implementations use a larger number, e.g. four samples). The reservoir is typically initialized to a sample drawn using a simple method and is then updated by RIS steps and by reservoir merging, so that the pixel value produced by shading using the sample(s) currently in the reservoir, times the weight for the sample, is always an unbiased estimate of the correct pixel value. If appropriate resampling steps are used, the variance of this estimate (or some function of it, typically the luminance of the RGB color value) decreases with each step. A possible sequence of steps performed for each frame, suitable for computing unbiased direct illumination (DI) is: Perform reservoir resampling by drawing multiple light samples and using streaming RIS to choose one, using probabilities based on a target function, e.g. the luminance of the sample's contribution to the pixel. A weight is also computed for the sample. Typically, a single visibility check is performed here, after choosing a sample, setting the weight to 0 if the light is shadowed. Resampling (combined with the visibility check) ensures that the expected value of the weight times the sample brightness is the correct (unbiased) value for the pixel. (temporal reuse) For each pixel, merge the sample(s) from the previous frame into the current reservoir. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in the previous frame's reservoirs may have a different target probability distribution if the objects, lights, or camera have moved. (spatial reuse) For each pixel, choose one or more neighboring pixels and merge their samples into the current pixel's reservoir. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in each pixel's reservoir have a different target probability distribution. Because computing unbiased MIS weights requires tracing additional rays (along with other work such as evaluating BRDFs), real-time rendering often uses only a single neighboring pixel. Use the sample in each pixel's reservoir, along with its weight, to determine the color of the pixel for the current frame. Alternatively, multiple samples examined during the preceding steps may be averaged and used to shade the pixel instead (decoupled shading and sampling). For direct lighting, the initial samples used in step 1 are typically drawn by importance sampling from the set of lights in a scene. The algorithm above (from the original ReSTIR paper) draws many lower-quality light samples (e.g. 32) using a fast method, without considering visibility, and chooses one using streaming RIS. Visibility is then tested for the final chosen sample. Considering visibility for each sample drawn would require tracing 32 rays, which would make it much more expensive. The intent is to reduce the number of rays traced, relying on the sample reuse in steps 2 and 3 to make up for the loss of quality caused by rejecting many of the rays due to shadowing. A large part of the initial efforts to optimize ReSTIR (to make it run in real-time on available hardware) went into reducing the cost of randomly sampling the lights. Glossy surfaces may require a larger number of samples, and combining light sampling with BRDF sampling (using MIS) may increase quality. Step 2 (temporal reuse) is sometimes skipped for off-line rendering, and the output of multiple repetitions of initial sampling and spatial reuse is averaged instead; this helps avoids artifacts due to correlations. Step 3 (spatial reuse) may be repeated multiple times in a single frame.
Data drilling
Data drilling (also drilldown) refers to any of various operations and transformations on tabular, relational, and multidimensional data. The term has widespread use in various contexts, but is primarily associated with specialized software designed specifically for data analysis. == Common data drilling operations == There are certain operations that are common to applications that allow data drilling. Among them are: Query operations: tabular query pivot query === Tabular query === Tabular query operations consist of standard operations on data tables. Among these operations are: search sort filter (by value) filter (by extended function or condition) transform (e.g., by adding or removing columns) Consider the following example: Fred and Wilma table (Fig 001): gender, fname, lname, home male, fred, chopin, Poland male, fred, flintstone, bedrock male, fred, durst, usa female, wilma, flintstone, bedrock female, wilma, rudolph, usa female, wilma, webb, usa male, fred, johnson, usa The preceding is an example of a simple flat file table formatted as comma-separated values. The table includes first name, last name, gender and home country for various people named fred or wilma. Although the example is formatted this way, it is important to emphasize that tabular query operations (as well as all data drilling operations) can be applied to any conceivable data type, regardless of the underlying formatting. The only requirement is that the data be readable by the software application in use. === Pivot query === A pivot query allows multiple representations of data according to different dimensions. This query type is similar to tabular query, except it also allows data to be represented in summary format, according to a flexible user-selected hierarchy. This class of data drilling operation is formally, (and loosely) known by different names, including crosstab query, pivot table, data pilot, selective hierarchy, intertwingularity and others. To illustrate the basics of pivot query operations, consider the Fred and Wilma table (Fig 001). A quick scan of the data reveals that the table has redundant information. This redundancy could be consolidated using an outline or a tree structure or in some other way. Moreover, once consolidated, the data could have many different alternate layouts. Using a simple text outline as output, the following alternate layouts are all possible with a pivot query: Summarize by gender (Fig 001): female flintstone, wilma rudolph, wilma webb, wilma male chopin, fred flintstone, fred durst, fred johnson, fred (Dimensions = gender; Tabular fields = lname, fname;) Summarize by home, lname (Fig 001): bedrock flintstone fred wilma Poland chopin fred usa ... (Dimensions = home, lname; Tabular fields = fname;) ==== Uses ==== Pivot query operations are useful for summarizing a corpus of data in multiple ways, thereby illustrating different representations of the same basic information. Although this type of operation appears prominently in spreadsheets and desktop database software, its flexibility is arguably under-utilized. There are many applications that allow only a 'fixed' hierarchy for representing data, and this represents a substantial limitation. == Drillup == Drillup is the opposite of drilldown. For example, if you drilldown to see the revenue of one product, then you might want to drillup to see the revenue of all products.