Integrated Public Use Microdata Series International
census microdata for social and economic research

IPUMS-United Kingdom: Proposal to Recover Microdata for the 1961-1981 Censuses
and Develop Anonymized, Integrated High Precision Household Samples

Note: no data are available from this site
Point and Click to view documentation

---

News Flash! 26-March-2010.
Letter from ONS Census Director
The ONS-UK "confirms support in principle"
to construct samples beginning with the 1961 census, as feasible

---

Proposal to recover and develop microdata of the 1981, 1971, 1966 and 1961 censuses of the UK
(for submission to the ONS; draft Sep. 19, 2008)

§ Click here for a PDF version of this document

Update Feb. 1, 2009.

1. National Science Foundation renewed IPUMS funding for a third five-year period—“top funding priority”—through 2013. UK historical census samples remains a top priority for IPUMS, as it has since 1999.

2. ULCC confirms continued eagerness to develop historical sample microdata with ONS approval.

3. Mark Elliott confirmed that, with ONS approval, he will design, implement, test and certify anonymization of sample for 1971 as a prototype. Once approved by ONS, refrined procedures would be applied to 1981 and 1961/66.

4. Ed Fieldhouse, on behalf of CCSR, has offered to assist to the extent feasible.

5. Peter Elias has offered his assistance in obtaining approvals from ONS.

Executive summary. The United Kingdom possesses one of the richest collections of individual level census data of all the countries in the world, stretching from 1851 to 2001. Nevertheless, the microdata for the censuses of 1961, 1966, 1971, and 1981, while secure, are not adequately archived, documented nor anonymized and thus are unavailable to researchers or the public. Working cooperatively in consortium, we propose to develop 4 datasets for each census, to the extent feasible:

1. “raw” microdata for future deposit in the VLM maintained by the ONS in cooperation with ESDS, UKDA, and CCSR

“Raw” samples deposited with the ESDS, UKDA and CCSR. 3 “user friendly” samples will be developed as resources and timetable permit, as follows: .

2. ULCC: Individual samples for UK I-SARs: greater geographic, less socio-demographic detail.

3. ULCC: Household samples for UK H-SARs (1991 model): less geographic, greater socio-demographic detail.

4. ULCC: International Household (IPUMS) sample derived from the UK H-SARs.

a. The variables in the IPUMS household sample will consist of a reduced set, different from the UK H-SAR sample. The IPUMS sample is for international researchers, interested in comparative research, while the H-SARs is aimed at UK users, whose research interests are likely to be national, regional or even local.

b. Anonymization of IPUMS samples is to be planned, performed and certified by the CCSR with ULCC support and ONS supervision--all before the microdata are entrusted to the IPUMS project. UKDA is invited to administer the project.

The ONS retains ultimate authority over the project as well as copyright over both microdata and metadata.

Timing. The timing for the initiative is propitious because government institutions are under increasing pressure to ensure that acesss to public data is guaranteed (www.freeourdata.org.uk/blog ). Earlier this year, the ESRC’s response to the Ministry of Justice regarding data sharing consultation makes the case for benefits from greater data sharing (http://www.justice.gov.uk/docs/Economic-and-Social-Research-Council-Data-Sharing-Response.pdf ). The National Data Strategy for Research in the Social Sciences strongly endorses the development of microdata sources in its 2007 report http://www2.warwick.ac.uk/fac/soc/nds/National_Data_Strategy_Final_tcm6-16581.pdf .

Eight years ago (June 2000), the ONS expressed support for granting permission for the development of the 1961-81 census microdata-- once the 2001 samples were launched. In 2006 and 2007, meetings were held at ONS to move the recovery project forward, and despite a clear, concise statement of next steps laid out by the ONS committee, no decision has been forthcoming as yet. Now, with ONS permission, work should begin without further delay because once operations for the next census are underway the ONS is not likely to wish to re-consider the issue for some time.

After the three datasets are developed, the IPUMS project would propose to launch integrated samples within two to three years.

Tasks. Given the great success of the 1991 SARs and the fact that the microdata to be recovered is over 25 years old, we propose to adopt the ’91 model within a collaboratory of Anglo-American institutions—ONS, UKDA, CCSR, ULCC, ESDS, and MPC. Each organization is invited to take responsibility for various tasks for work to be undertaken and completed in 2009 with funding provided by the IPUMS-International project and from resources contributed by each institution.

Budget. Although population censuses cost hundreds of millions of pounds to carry through, the marginal costs of recovering historical census microdata is trivial, ranging, in the IPUMS-International experience, from £2-10,000 per census (for France and Netherlands, respectively—these are in-country costs charged by national statistical offices and/or seconded academic institutions or individual consultants working as deemed employees).

Proposed Budget for Historical Census Microdata Pilot Project
Recovery of 1971 Census Microdata and Construction of Anonymized Household Sample
		Budget (£)
Task	Institution	Senior Person(s)	Junior Person(s)
1. Administration	UKDA	1000	800
2. Anonymization plan	CCSR	2500	None
3. Recover microdata and metadata	ULCC	13000	18000
4. Archive microdata and metadata	UKDA, CCSR	500	800
5. Draw Non-overlapping Individual and Household Samples	ULCC	4000	4800
International Household Sample only (development of UK samples is the responsibility of UK institutions)
1. Design Suppression of Variables and Codes	IPUMS, CCSR	1000	None
2. Weight records	ULCC	2000	1600
3. Suppress and anonymize	ULCC	4000	4800
4. Certify anonymization	CCSR, ONS	2500	None
5. Integrate household sample into IPUMS	IPUMS	-	-
Funding provided by IPUMS-International	Totals	30500	30800

Collaboration. The UKDA is invited to assume overall administration of the UK census microdata recovery project. The UKDA is already cooperating with the IPUMS-International project, specifically with the preservation of the entire stock of census microdata entrusted to IPUMS—approximately 7 terrabytes, more than 200 censuses encompassing more than 70 countries.

The ULCC has already invested considerable resources in archiving the microdata and metadata for the historical censuses. With ONS permission, the ULCC is eager to complete the process beginning with the 1971 census as a pilot, and, if successful, continue with the recovery of microdata and metadata of other UK historical censuses..

The CCSR prepared the anonymization protocols for both the 1991 and 2001 SARs, and has agreed to develop and implement an anonymization plan for the historical censuses, subject to ONS granting permission. In addition, the CCSR would provide results of certification tests to the ONS for review, prior to entrusting microdata to the IPUMS-International project.

The IPUMS-International collaboratory (https://international.ipums.org/international) has recovered microdata for many countries from the 1960 census round forward. Anonymized, integrated microdata samples for 35 countries are currently available, on a restricted access basis, to researchers world-wide without cost. The database encompasses 111 censuses, 72 million households, and 263 million person records, including for the UK the 1991 H-SARs and the 2001 I-SARs. The National Science Foundation and the National Institutes of Health USA provides major funding for IPUMS.

The IPUMS-Europe initiative offers two prototypes for recovering census microdata and developing samples (including anonymization and the assembly of documentation):.

1. Performed entirely by the national statistical office under contract to IPUMS.

§ Statistics Netherlands. 1960 and 1971 and a register based sample for 2001—according to IPUMS specifications but all worked performed by SN..

§ Federal Statistical Office—Germany: All work performed by FSO to IPUMS specifications, including the 1970 and 87 censuses of the FRG and the 1971 and 81 censuses of the GDR

§ Federal Statistics Office—Switzerland: 1971, 1981, 1991, and 2001 – prepared by the FSO to IPUMS specifications.

2. Seconded to an institution or individual under supervision of the national statistical authority

§ INSEE—France: 1962, 1968, 1975, 1982 and 1990 –prepared by an individual researcher working within the INSEE under contract with IPUMS and with INSEE oversight.

§ INSSE—Romania: Work performed by a university research institute for the censuses of 2002, 1992, and 1977 under contract with IPUMS and with INSSE oversight.

Proposed pilot: 1971 census.

Coordination: UKDA

Conduct the test in three steps:

a. ULCC: Recover the census microdata and corresponding documentation

i. Tape images have already been transferred to modern media and record blocks have been recovered.

ii. Complete the documentation of data dictionaries and codebooks

iii. Document the sample design of source microdata to the extent possible.

iv. Test the correspondence of microdata frequencies with official figures, documenting any geographical units that may be omitted, variables that are corrupted.

v. Determine whether national representative samples may be drawn from recovered microdata..

b. ULCC: Where feasible, draw and document individual and household samples. Both are to be drawn with implicit geographical stratification, consisting of every n^th person or household. Both samples are entrusted to ONS, UKDA, CCSR, and ESDS

i. For confidentiality purposes the individual sample will have greater geographic detail, and less socio-demographic

ii. There will be no overlap between the samples. If, for the hierarchical (household) sample, a sample point includes a person in the individual sample, then the household immediately following of the same size is to be substituted.

iii. Include Northern Ireland and Scotland, if permission is granted

iv. For communal households (or private households with x persons or more (31 is the IPUMS standard), draw every n^th person in hh (attach dwelling and household characteristics to person)

v. Confirm validity of samples (compare with published totals; note there will be some deviations due to use of household as sample unit)

vi. Prepare comprehensive data dictionary and codebooks

vii. Provide copies of enumerator manuals (instructions on completing questionnaires).

c. ULCC, CCSR, IPUMS: Household sample to be entrusted to IPUMS-International

i. IPUMS: suppress variables and codes that are not appropriate for the internationally harmonized sample.

ii. CCSR: Confidentialize the sample using the 1991 SARs as a point of departure

Once approved, apply same procedures to 1981, 1966 and 1961. Where only sample microdata survive for a census, use the entire surviving dataset for step “a”.

Addendum. To access a document, click its link.

Proposal to recover and develop historical microdata for the censuses of 1961-1981, to the extent feasible.

§ PDF version of this plan

§ 1991 H-SARs variables (see Tables 1 and 2, below or click the link):
List of 18 household and 24 person variables provided to IPUMS
and the 9 household and 18 person variables integrated into IPUMS

§ Listing of 500+ Documents for the 1961-1981 Censuses of the UK
scanned by the IPUMS projects

§ Census Forms—to illustrate the correspondence in the questions from census to census)

o 1961

o 1966

o 1971

o 1981

o 1991

H-SARs 1991 (questions on forms of 1961-1981 censuses are most consistent with 1991)

§ H-SARs documentation (see 1991)

§ H-SARs Disclosure Control Methods

IPUMS-International project documents

Production web site: https://international.ipums.org/international

Work in progress (Bob’s web site): http://www.hist.umn.edu/~rmccaa/IPUMSI

Background

§ List of census microdata entrusted by country and census year

§ IPUMS-International National Statistical Office partners

§ Archiving microdata and metadata (UNSD, 2001)

§ Census microdata revolution (2002)

§ Project overview

§ Integration methods

§ Memorandum of understanding with National Statistical Office partners

Statistical confidentiality and managing access to microdata

§ The Trewin Report, an on-site inspection of IPUMS-International (Nov. 2007)

§ Statistical Confidentiality: IPUMS-Europe principals and practices (Geneva 2006)

§ High precision samples/restricted access extracts (PSD 2006)

§ UN-ECE IPUMS case study of good practice: Managing Statistical Confidentiality and Microdata Access (Geneva 2007)

Using IPUMS

§ Users and Uses of IPUMS-International microdata (ANCSDAAP conference, 2007)

§ Tips on accessing IPUMS-International census microdata

§ Tutorial: Dynamic metadata system

Notes of meetings with representatives of the Office of National Statistics
(discussions of 1991 and 2001 samples may be ignored as no longer pertinent)

§ June 2000, London: Chris Denham, Danny Dorling, Bob McCaa, Humphrey Southall, Angela Dale, Kevin Ashley

§ June 2006, Titchfield: Bob McCaa, Laurel Bran, Simon King, Keith Spicer

§ Oct 2006, Titchfield: Bob McCaa, Laurel Bran, Jane Longhurst, Keith Spicer

Table 1. H-SARs 1991 Variables entrusted to IPUMS-International
	Household Variables	No. of Codes	Comments with reference to 1961-1981 census microdata
1	Dwelling number	constructed
2	Number of persons in household	11
3	Number of household spaces in dwelling	2
4	Household space dwelling type	14
5	SAR region	13	N. Ireland omitted; include for historical census microdata
6	Ward group codes	27	not useful for international comparison; better would be size of place variable
7	Tenure of household space	10
8	Central heating	3
9	Number of cars	4
10	Bath/shower	3	98% in categoriy 1
11	Availability of inside flush toilet	3	99% in category 1
12	Lowest floor level of accommodation	8
13	Number of rooms in household space	15
14	Wholly moving household	3	90% in category 2
	Person Variables
1	Person number within household	11
2	Age	94
3	Family head	2	constructed by CCSR?
4	Marital status	5	only 4 categories available before 1991
5	Relationship to head of household	17
6	Sex	2
7	Country of birth	42
8	Number of higher qualifications	3
9	Level of highest qualification	4
10	Subject of highest qualification	90	12 subject groups
11	Distance to work	11
12	Economic position (primary)	11
13	Economic position (secondary)	11
14	Class of worker	6
15	Usual hours worked	74
16	Mode of transport to work	11
17	Workplace	7
18	Occupational classification	360	79 sub-minor, 24, submajor and 11 major
19	Industrial classification	187	118 3 digit; 31 2 digit; 11 1 digit; 13 sic divisions
20	Distance of move (migrants)	16
21	Region of residence 1 year ago	15
22	Resident status	3
23	Visitor: area of usual residence	15
24	Family number	5
	Variables Constructed by CCSR
1	Household composition type	23	constructed by CCSR
2	Household dependent type	23	constructed by CCSR
3	Class schema used in women and employment survey	14
4	Standard international occupational scale	46
5	Mean hourly earning derived from New Earnings Survey, 1991
6	Mean earnings from New Earnings Survey, for previously working persons
7	Social class (based on occupation)
8	Socio-economic group
9	Goldthorpe class
10	Lifestages classification
11	New Earning Survey standard error
12	International socio-economic index of occupational status
13	Family type	9
14	Type of minimal household unit	10
15	Position in minimal household unit	16
16	Combinations of minimal household units	26
	Variables in 1991 SARs but not in 1961-1981 censuses
1	Ethnic group	10
2	Gaelic language	7
3	Welsh language	7
4	Term-time address	5
5	Limiting long-term illness	2

Table 2. 1991 H-SARs variables integrated into IPUMS-International
No	IPUMS-International Variable Label	IPUMS Codes UK 1991	No. of
			Codes
	Household Variables
1	Group quarters status	codes	2
2	Number of unrelated persons	codes	10
3	Region, United Kingdom	codes	16
4	Ownership of dwelling	codes	2
5	Automobiles available	codes	3
6	Central heating	codes	2
7	Number of rooms	codes	14
8	Toilet	codes	2
9	Number of families in household	codes	9
	Person Variables
1	Person number	codes	11
2	Relationship to household head	codes	17
3	Age	codes	92
4	Sex	codes	2
5	Marital status	codes	4
6	Nativity status (native born/foreign)	codes	2
7	Country of birth	codes	35
8	Educational attainment, United Kingdom	codes	4
9	Employment status	codes	7
10	Occupation, unrecoded	codes	3 digit
11	Industry, unrecoded	codes	3 digit
12	Class of worker	codes	6
13	Full-time or part-time work	codes	3
14	Hours worked in main occupation	codes	74
15	Means of transportation to work or school	codes	12
16	Migration status, 1 year	codes	5
17	Region of residence 1 year ago, United Kingdom	codes	15
18	Employment disability	codes	3
	Constructed Variables
1	Relationship to head, Europe	codes	derived
2	Employment status, Europe	codes	derived
3	Occupation, ISCO	codes	derived
4	Industry, general recode	codes	derived
5	Class of worker, Europe	codes	derived
6	NUTS1 Region, Europe	codes	12