Internal Security

Overview

Internal Security in Ilum is an internal authorization and authentication system managed by Ilum Core. It securely stores all user data in a database and is used within the Ilum Core Server environment.

Ilum’s internal security system provides Role-Based Access Control (RBAC), allowing every action on the Ilum Core Server to be mapped to a specific permission. This enables highly precise access management.

To simplify permission assignments, you can create user roles and groups that define the permissions for users associated with these roles or groups.

Example

Let’s create a user who can access the existing data architecture to retrieve and analyze data but does not have permission to modify it by adding new pods.

The first step is to create a role. This helps streamline the process by avoiding the need to repeatedly assign the same permissions to multiple users.

1. Navigate to Security -> Roles and click the Add Role button.
2. Fill in the role details, including the name and description. Remember to check the Enabled checkbox to activate the role.

3. Next, go to Permissions and assign the appropriate permissions for the role.

For this example, we will configure permissions based on the following assumptions:

• The user should have full access to information about the data architecture to utilize it effectively. Assign Read permissions for the following resources: Jobs, Services, Schedules, Clusters, and Job Requests.
• The user should be able to work within the data architecture to transform data without altering critical parts of the system. Assign Execute permission for the Service resource. Assign Create permission for the Job resource.
• The user should have access to Ilum components such as File Explorer, Lineage, Table Explorer, History Server, Superset, Jupyter or Zeppelin Notebooks, and Ilum SQL. Grant Read permission to use each of these components.

4. The user should be aware of their privileges but not have the ability to modify them. Assign Read permissions for Groups, Roles, and Users

After creating the role, the next step is to create the user.

1. Navigate to Security -> Users and click the Add User button.
2. Fill in the user details and enable the account by checking the Enabled checkbox.

3. Next, assign permissions to the user.

The easiest way to do this is by assigning the role you created earlier. This will automatically grant all the desired permissions to the user.

If additional permissions are required beyond those in the role, you can assign them directly to the user as needed.

Finally, the user is ready. You can log in into application using its username and password.

Access control

Each action in Ilum Core requires a specific permission. As of the time of writing this article, there are 48 different permissions that can be assigned. To simplify access control, it is recommended to use groups and roles.

Groups represent collections of users within your organization. By assigning a user to a group, that user automatically inherits all the permissions associated with the group. Group permissions can be granted both directly, as well as through roles assigned to the group.

A user can belong to multiple groups and have several roles assigned to them. This means the user will inherit all the permissions granted by those groups and roles. Additionally, you can assign permissions directly to the user. Ilum Core will then check the permissions by considering those directly assigned to the user, as well as those from the groups and roles they belong to, including roles assigned to the groups.

Note: You can disable roles and groups, which will prevent users from inheriting permissions from them. Similarly, disabling users will prevent them from accessing anything within the application.

Permissions

Each action in Ilum Core requires it's own permission.

1. Ilum Job permissions:

• JOB_READ
• JOB_CREATE
• JOB_DELETE - to delete jobs
• REQUEST_READ

Note: JOB_READ permission gives you access to read everything about job: its error and success message, its logs, its overview with metrics etc. REQUEST_READ permission gives you access to read a list of all requests and details of each one of them

2. Ilum Service permissions:

• SERVICE_CREATE
• SERVICE_DELETEARCHIVE
• SERVICE_EDIT
• SERVICE_PAUSERESUME
• SERVICE_EXECUTE
• SERVICE_READ

3. Schedule permissions:

• SCHEDULE_READ
• SCHEDULE_CREATE
• SCHEDULE_EDIT
• SCHEDULE_DELETE
• SCHEDULE_PAUSERESUME

4. Cluster permissions:

• CLUSTER_READ
• CLUSTER_EDIT
• CLUSTER_CREATE
• CLUSTER_DELETE

Note: CLUSTER_READ does not give access to read services, schedules and jobs inside of a cluster. SERVICE_READ, SCHEDULE_READ and JOB_READ is required

5. Modules access.

Each module requires a permission to access it. This permission gives full access to the module.

• NOTEBOOK_READ: Jupyter and Zeppelin
• LINEAGE_READ
• SUPERSET_READ
• HISTORYSERVER_READ
• MINIO_READ
• AIRFLOW_READ
• MLFLOW_READ
• SQL_READ
• TABLEEXPLORER_READ
• FILEEXPLORER_READ

6. Security management

These are permissions required to observe, update, delete, enable and disable users, roles and groups. In addition, there are permissions to observe activities history and clear it.

• USER_CREATE
• USER_EDIT_PASSWORD (to reset user password)
• USER_EDIT_DETAILS (to edit user name, description, fullname, department, email)
• USER_EDIT_ACCESS (to edit user permissions, roles and groups)
• USER_DELETE
• USER_ENABLEDISABLE
• USER_READ
• USERROLE_CREATE
• USERROLE_READ
• USERROLE_DELETE
• USERROLE_EDIT
• USERROLE_ENABLEDISABLE
• GROUP_READ
• GROUP_DELETE
• GROUP_CREATE
• GROUP_EDIT
• GROUP_ENABLEDISABLE
• ACTIVITY_READ
• ACTIVITY_DELETE

Default roles

By default Ilum Core offers 4 roles to you:

1. ADMIN: Has access to all permissions
2. USER: Has access only to read permission except for MLFlow, Minio, Ilum sql, Airlfow and Notebooks. Has USER_EDIT_DETAILS permission
3. DATA_ENGINEER: Has access to all permissions except for security permissions. Has USER_EDIT_DETAILS permission
4. DATA_SCIENTIST: Has all read permissions except for MLFlow and Airflow. Has SERVICE_EXECUTE and JOB_CREATE permissions. Has USER_EDIT_DETAILS permission

Activities

Ilum maintains a history of user operations, which can be viewed in the Security -> Activities folder. Here, you can see a brief description of each action performed, along with the endpoint used, all associated path variables and its timestamp.

You can filter the activity log by name or by endpoint to quickly find relevant information.

Deployment

Ilum offers 3 security types and you can choose only one of them at the time. In order to choose internal security you should set corresponding helm configurations to internal as below:

helm upgrade ilum ilum/ilum \\
--set ilum-core.security.internal.enabled=true \\
--set ilum-core.security.type=internal --reuse-values

Additionally, you can use Helm configuration to add initial users to your application. Users added via Helm configuration will be created during the application startup and labeled as Root Users.

Root Users come with certain restrictions: You cannot delete, disable, change their username, or restrict their access. However, you can update their details, such as their full name, email, description, or reset their password. This ensures that you always have guaranteed access to your application.

If you need to update the initial users, simply modify your Helm configuration. Ilum will automatically apply the updated configuration and update the Root Users on the next application startup.

You can add users in helm with this configuration.yaml:

ilum-core: 
  security: 
    internal:
      users:
        - username: "admin2"
          password: "admin"
          roles:
            - "ADMIN"
        - username: "user2"
          password: "admin"
          roles: 
            - "USER"
          

helm upgrade ilum ilum/ilum -f configuration.yaml --reuse-values

You can configure all fields of a user in helm like in the example below:

ilum-core: 
  security: 
    internal:
      users:
      - username: "myuser"
        email: "email"
        fullname: "fullname"
        department: "department"
        description: "description"
        password: "password"
        roles: 
          - "ROLE1"
          - "ROLE2"
          - "ROLE3"
        permissions: 
          - "SERVICE_CREATE"
          - "SERVICE_EXECUTE"
        groups: 
          - "groupId1"
        state: ENABLED
        rootState: false

Note: in case you don't want your initial user to be a root user, you can set rootState flag to false