Anonymisation

Mudlark can now be used to fully anonymise a Maintenance Work Order CSV file. In this section we will cover anonymising the text column, and anonymising other columns.

Anonymising text

To anonymise text, simply set the anonymise_text argument to True, either via command line or in a config file.

When this is set to true, any terms that Mudlark deems to be “asset identifiers” will be anonymised and replaced with a unique asset identifier. For example, given the following data:

text
ABC 124 is broken
ABC 123 has a problem
ABC-124 is broken
pumps busted
enGiNe was broken
a leak was Formed

The result will be:

text
Asset1 is broken
Asset2 has a problem
Asset1 is broken
pump bust
engine is broken
a leak is form

Note how both ABC 124 and ABC-124 both become Asset1 and ABC 123 becomes Asset2.

Some further notes on how this works:

  • Currently a term is recognised as an asset identifier if it has one or more uppercase letter(s), followed by either nothing, a space, or a hyphen, then followed by one or more digit.

  • The order is randomised prior to generating the numbers for each asset identifier.

Anonymising/processing other columns

Mudlark can also anonymise and/or process other columns. To do this, set column_config_path prior to running Mudlark (either via command line or in a config file). An example column config file might look like this:

columns:
  - name: floc
    handler: FLOC
  - name: mwo_id
    handler: RandomiseInteger
  - name: unique_floc_desc
    handler: ToUniqueString
    prefix: FLOC_Desc_
  - name: other
    handler: None

This file dictates which columns should be kept by Mudlark, and how it should handle each of them. There are currently four available “handlers”:

  • None simply passes the column through to the output file without doing anything to it.

  • RandomiseInteger replaces each unique value of the column with a randomised integer (with 7 digits).

  • FLOC treats the column as a Functional Location. It splits it based on either a . or a -, then converts each unique value of each level of the FLOC hierarchy into a number. For example, 123-45-67 might become 1-1-1, 123-45-68 might become 1-1-2, and so on.

  • ToUniqueString converts each unique value into an anonymised string, starting with the given prefix. For example, Pump FLOC might become FLOC_Desc_1, Belt FLOC might become FLOC_Desc_2, and so on.

Example

Here is an example dataset:

text,cost,other,floc,mwo_id,unique_floc_desc
ABC 124 is broken,123,test,123.45.67,123,FLOC 123
replace,43,xx,123.45.68,123,FLOC 124
X/X,540,test,123.45.69,123,FLOC 125
ABC 123 has a problem,3,test,123.45.67,123,FLOC 123
slurries,4.3,xx,123.45.67,123,FLOC 123
ABC-124 is broken,4.33,yrds,123.45.67,123,FLOC 123
pumps busted,43.43,tyrdtrd,111.45.67,123,FLOC 125
enGiNe was broken,4332.3,6t554,112.45.67,123,FLOC 126
a leak was Formed,333,545,113.45.67,123,FLOC 127

We are going to anonymise the text (as discussed in the first section), and will keep the floc, mwo_id, and unique_floc_desc columns. Here is our column-config.yml:

columns:
  - name: floc
    handler: FLOC
  - name: mwo_id
    handler: RandomiseInteger
  - name: unique_floc_desc
    handler: ToUniqueString
    prefix: FLOC_Desc_
  - name: other
    handler: None

And our Mudlark config file:

input_path: my-file.csv
output_path: my-file-out.csv
text_column: text
output_format: csv
anonymise_text: true
column_config: column-config.yml

Once we run Mudlark, the output will be:

text,other,floc,mwo_id,unique_floc_desc
Asset1 is broken,test,1_1_1,2462749,FLOC_Desc_1
replace,xx,1_1_2,7832383,FLOC_Desc_2
x/x,test,1_1_3,5472030,FLOC_Desc_3
Asset2 has a problem,test,1_1_1,2806910,FLOC_Desc_1
slurry,xx,1_1_1,1640112,FLOC_Desc_1
Asset1 is broken,yrds,1_1_1,7360650,FLOC_Desc_1
pump bust,tyrdtrd,2_1_1,9995977,FLOC_Desc_3
engine is broken,6t554,3_1_1,6573352,FLOC_Desc_4
a leak is form,545,4_1_1,6717645,FLOC_Desc_5