A human-machine collaboration to defend against cyberattacks

http://news.mit.edu/2020/patternex-machine-learning-cybersecurity-0221

Basic concepts of docker/kubernete/kata-container

Kubereters

An open-source system for automating deployment, scaling, and management of containerized applications;

Traditional >> Hypervisor >> Container

Traditional:

　　No way to define resource boundaries for applications in a physical server before, run each applications on a different physical server;

Hypervisor: 

　　Will run multi VMs on a single physical server''s CPU; 

　　Allow applications to be isolated between VMs and provides a level of securiy as the infomation of one application cannot be freely accessed by another 　　　　application;

　　Each VM is a full machine running all the components, including its own operating system;

Container:

　　Have relaxed isolation properties to share the operating system among the applications;

　　Lightweight;

　　A container has its own filesytem/CPU/memory/process space/..

　　Are portable across clouds and OS distributions because they are decoupled from the underlying infrastrucure;

　　Benifits of containers:

　　　　1. Agile application creation and deployment: increased ease and efficiency of container image creation compared to VM image use

　　　　2. Continuous development, integrations and deployment:

　　　　　　provide for reliable and frequent container image build and deployment with quick and easy rollbacks;

　　　　3. Dev and Ops separation of concerns: create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure;

　　　　4. Obervability not only surfaces OS-level information and metrics, but also application health and other signals;

　　　　5. Environmental consistency across development, testing, and production; Runs the same on a laptop as it does in the cloud;

　　　　6. Cloud and OS distribution portabilty: Runs on Ubuntu, RHEL, CoreOS, on-prem, Google Kubernetes Engine, and anywhrere else;

　　　　7. Application-centric management;

　　　　8. Loosely coupled, distributed;

　　　　9. Resource isolation;

　　　　10. Resource utilization;

Benefits of Kubernetes:

　　1.  Service discovery and load balancing　　　　

　　　　　　kubernetes expose a container using the DNS name or using their own IP address;

　　　　　　if traffic to a container is high, kubernetes is able to load balance and distribute the network traffic so that the deployment is stable;

　　2.  Stroage orchestartion

　　　　　　automatically mount a storage system of local storages/public cloud providers/..

　　3.  Automated rollouts and rollbacks

　　　　　　automake kubernets to create new containers

　　4.  Self-healing

　　　　　　restart fail containers, kill no-respond containers;

　　5.  Automatic bin packing

　　　　　　tell kubernetes how much CPU and RAM each container needs, kubernets can fit containers onto your nodes to make the best use of you resources;

Kubernetes Master

When deploying kubernetes, we wil get a cluster, which is a set of machines (nodes),

that run containerzed applications managed by kubernetes.

A cluster has at least one worker node and at least one master node;

Kubernetes Master is a collection of three processes that run on a single node in your cluster, which is designed as the master node;

Three processes:

　　Kube-apiserver;

　　Kube-controller-manager;

　　Kube-scheduler;

Individual non-master node in cluster run two processes:

　　kubelet

　　　　which communicates with the Kubernetes Master;

　　　　The primary node agent that runs on each node; It can register the node with tha apiserver using one of

　　　　　　the host name;

　　　　　　a flag to override the hostname;

　　　　　　specific logic for a cloud provider;

　　kube-proxy

　　　　a network proxy which reflects Kubernetes networking services on each node;

　　　　Kubernets network proxy runs on each node, this relects services as defined in the Kubernetes API on each node

　　　　and can do simple TCP/UDP and SCTP stream forwarding or round robin TCP/UDP/SCTP forwarding across a set of backends/

　　　　Service cluster IPs and ports are currently found through Docker-links-compatible　　　

Kubernetes Objects

Kubernetes contains serveral abstractions representing the state of system;

Basic Kubernetes objects include:

• Pod
• Service
• Volume
• Namespace

Kata-container

An open-source project and community working to build a standard implement of lightweight VM that feel and perform like containers, but provide the workload isolation and security advantages of VMs;

Kata container Components

• Agent　　　　　　-- The Kata-agent runs inside the virtual machine and sets up the container environment
• KSM throttler　　 -- An optional utility that monitors containers and deduplicates memory to max container density on a host
• Proxy　　　　　　-- A process running on the host and co-ordinates access to the agent running inside the VM
• Runtime　　　　　-- Be invoked by a container manager and provides high-level verbs to manage containersd  
• Shim　　　　　　  -- A process that runs on the host, acts as though it is the workload ( which actually runs inside the VM), required to be compliant with th expecations of the OCI runtime sepc
• Hypervisor　　　　-- 
• Kernel　　　　　　-- HV uses a linux kernel to boot the guest image

Docker

Following storage drivers:

• overlay2 　　　　is the preferred storage driver, for all currently supported linux distributions, and requires no extra conf
• aufs 　　　　　　is the preferred stroage driver for Docker 18.06 or older, when running on Ubuntu 14 on Kernel 3.13 which has no support or ovrlay2
• devicemapper 　 is supported, but requires direct-lvm for production environments, because loopback-lvm, while zero-conf, has very poor performance.

centos rancher 通过本机 docker images 新增 container

示例目标： 将 centos 本地的 docker image ，通过 rancher 进行 add container

相关步骤：

step 1 ： rancher 所在 centos 主机，有相应的 docker images ，如 testapi:v1

step 2 ：浏览器访问 rancher ui  -> infrastructure -> Containers -> Add Container

step 3 ： 输入 Name ， Select Images = testapi:v1 ，进行 create 保存

step 4 ： 会出现类似如下错误提示 

               Error (Image [index.docker.io/testapi:v1] failed to pull: repository docker.io/testapi not found: does not exist or no pull access)

step 5 ： 回到 centos 主机，通过 docker tag 进行类似如下处理 （模拟已将 docker.io/testapi 下载到本机的效果）

               #docker tag testapi:v1 docker.io/testapi:v1

               进行本地镜像查看

              #docker images

step 6 ： 回到 rancher ui, 删掉刚才添加失败的 testapi 容器，

              重新进行如 step 3 的添加， 

               这次就不会再出现 step 4 的 error。 

rancher 直接从本地镜像创建容器完成。

供参考

Configuring and Running Django + Celery in Docker Containers

Configuring and Running Django + Celery in Docker Containers

After reading this blog post, you will be able to configure Celery with Django, PostgreSQL, Redis, and RabbitMQ, and then run everything in Docker containers.

Today, you''ll learn how to set up a distributed task processing system for quick prototyping. You will configure Celery with Django, PostgreSQL, Redis, and RabbitMQ, and then run everything in Docker containers. You''ll need some working knowledge of Docker for this tutorial, which you can get in one my previous posts here.

Django is a well-known Python web framework, and Celery is a distributed task queue. You''ll use PostgreSQL as a regular database to store jobs, RabbitMQ as message broker, and Redis as a task storage backend.

Motivation

When you build a web application, sooner or later you''ll have to implement some kind of offline task processing.

Example:

Alice wants to convert her cat photos from .jpg to .png or create a .pdf from her collection of .jpg cat files. Doing either of these tasks in one HTTP request will take too long to execute and will unnecessarily burden the web server - meaning we can''t serve other requests at the same time. The common solution is to execute the task in the background - often on another machine - and poll for the result.  

A simple setup for an offline task processing could look like this:

1. Alice uploads a picture.  
2. Web server schedules job on worker.  
3. Worker gets job and converts photo.  
4. Worker creates some result of the task (in this case, a converted photo).  
5. Web browser polls for the result.  
6. Web browser gets the result from the server.  

This setup looks clear, but it has a serious flaw - it doesn''t scale well. What if Alice has a lot of cat pictures and one server wouldn''t be enough to process them all at once? Or, if there was some other very big job and all other jobs would be blocked by it? Does she care if all of the images are processed at once? What if processing fails at some point?

Frankly, there is a solution that won''t kill your machine every time you get a bigger selection of images. You need something between the web server and worker: a broker. The web server would schedule new tasks by communicating with the broker, and the broker would communicate with workers to actually execute these tasks. You probably also want to buffer your tasks, retry if they fail, and monitor how many of them were processed.

You would have to create queues for tasks with different priorities, or for those suitable for different kinds of workers.

All of this can be greatly simplified by using Celery - an open-source, distributed tasks queue. It works like a charm after you configure it - as long as you do so correctly.

How Celery is built

Celery consists of:

• Tasks, as defined in your app
• A broker that routes tasks to workers and queues
• Workers doing the actual work
• A storage backend

You can watch a more in-depth introduction to Celery here or jump straight to Celery''s getting started guide.

Your setup

Start with the standard Django project structure. It can be created with django-admin, by running in shell:

$ django-admin startproject myproject

Which creates a project structure:

.
└── myproject
    ├── manage.py
    └── myproject
        ├── __init__.py
        ├── settings.py
        ├── urls.py
        └── wsgi.py

At the end of this tutorial, it''ll look like this:

.
├── Dockerfile
├── docker-compose.yml
├── myproject
│   ├── manage.py
│   └── myproject
│       ├── celeryconf.py
│       ├── __init__.py
│       ├── models.py
│       ├── serializers.py
│       ├── settings.py
│       ├── tasks.py
│       ├── urls.py
│       ├── views.py
│       └── wsgi.py
├── requirements.txt
├── run_celery.sh
└── run_web.sh

Creating containers

Since we are working with Docker 1.12, we need a proper Dockerfile to specify how our image will be built.

Custom container

Dockerfile

# use base python image with python 2.7
FROM python:2.7

# add requirements.txt to the image
ADD requirements.txt /app/requirements.txt

# set working directory to /app/
WORKDIR /app/

# install python dependencies
RUN pip install -r requirements.txt

# create unprivileged user
RUN adduser --disabled-password --gecos '''' myuser  

Our dependencies are:

requirements.txt

Django==1.9.8  
celery==3.1.20  
djangorestframework==3.3.1  
psycopg2==2.5.3  
redis==2.10.5  

I''ve frozen versions of dependencies to make sure that you will have a working setup. If you wish, you can update any of them, but it''s not guaranteed to work.

Choosing images for services

Now we only need to set up RabbitMQ, PostgreSQL, and Redis. Since Docker introduced its official library, I use its official images whenever possible. However, even these can be broken sometimes. When that happens, you''ll have to use something else.

Here are images I tested and selected for this project:

• Official PostgreSQL image
• Official Redis image
• Official RabbitMQ image

Using docker-compose to set up a multicontainer app

Now you''ll use docker-compose to combine your own containers with the ones we chose in the last section.

docker-compose.yml

version: ''2''

services:  
  # PostgreSQL database
  db:
    image: postgres:9.4
    hostname: db
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "5432:5432"

  # Redis
  redis:
    image: redis:2.8.19
    hostname: redis

  # RabbitMQ
  rabbit:
    hostname: rabbit
    image: rabbitmq:3.6.0
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5672:5672"  # we forward this port because it''s useful for debugging
      - "15672:15672"  # here, we can access rabbitmq management plugin

  # Django web server
  web:
    build:
      context: .
      dockerfile: Dockerfile
    hostname: web
    command: ./run_web.sh
    volumes:
      - .:/app  # mount current directory inside container
    ports:
      - "8000:8000"
    # set up links so that web knows about db, rabbit and redis
    links:
      - db
      - rabbit
      - redis
    depends_on:
      - db

  # Celery worker
  worker:
    build:
      context: .
      dockerfile: Dockerfile
    command: ./run_celery.sh
    volumes:
      - .:/app
    links:
      - db
      - rabbit
      - redis
    depends_on:
      - rabbit

Configuring the web server and worker

You''ve probably noticed that both the worker and web server run some starting scripts. Here they are (make sure they''re executable):

run_web.sh

#!/bin/sh

# wait for PSQL server to start
sleep 10

cd myproject  
# prepare init migration
su -m myuser -c "python manage.py makemigrations myproject"  
# migrate db, so we have the latest db schema
su -m myuser -c "python manage.py migrate"  
# start development server on public ip interface, on port 8000
su -m myuser -c "python manage.py runserver 0.0.0.0:8000"  

run_celery.sh

#!/bin/sh

# wait for RabbitMQ server to start
sleep 10

cd myproject  
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
su -m myuser -c "celery worker -A myproject.celeryconf -Q default -n default@%h"  

The first script - run_web.sh - will migrate the database and start the Django development server on port 8000. 
The second one , run_celery.sh, will start a Celery worker listening on a queue default.

At this stage, these scripts won''t work as we''d like them to because we haven''t yet configured them. Our app still doesn''t know that we want to use PostgreSQL as the database, or where to find it (in a container somewhere). We also have to configure Redis and RabbitMQ.

But before we get to that, there are some useful Celery settings that will make your system perform better. Below are the complete settings of this Django app.

myproject/settings.py

import os

from kombu import Exchange, Queue


BASE_DIR = os.path.dirname(os.path.dirname(__file__))

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = ''megg_yej86ln@xao^+)it4e&ueu#!4tl9p1h%2sjr7ey0)m25f''

# SECURITY WARNING: don''t run with debug turned on in production!
DEBUG = True  
TEMPLATE_DEBUG = True  
ALLOWED_HOSTS = []

# Application definition

INSTALLED_APPS = (  
    ''rest_framework'',
    ''myproject'',
    ''django.contrib.sites'',
    ''django.contrib.staticfiles'',

    # required by Django 1.9
    ''django.contrib.auth'',
    ''django.contrib.contenttypes'',

)

MIDDLEWARE_CLASSES = (  
)

REST_FRAMEWORK = {  
    ''DEFAULT_PERMISSION_CLASSES'': (''rest_framework.permissions.AllowAny'',),
    ''PAGINATE_BY'': 10
}

ROOT_URLCONF = ''myproject.urls''

WSGI_APPLICATION = ''myproject.wsgi.application''

# Localization ant timezone settings

TIME_ZONE = ''UTC''  
USE_TZ = True

CELERY_ENABLE_UTC = True  
CELERY_TIMEZONE = "UTC"

LANGUAGE_CODE = ''en-us''  
USE_I18N = True  
USE_L10N = True

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.7/howto/static-files/
STATIC_URL = ''/static/''

# Database Condocker-composeuration
DATABASES = {  
    ''default'': {
        ''ENGINE'': ''django.db.backends.postgresql_psycopg2'',
        ''NAME'': os.environ.get(''DB_ENV_DB'', ''postgres''),
        ''USER'': os.environ.get(''DB_ENV_POSTGRES_USER'', ''postgres''),
        ''PASSWORD'': os.environ.get(''DB_ENV_POSTGRES_PASSWORD'', ''postgres''),
        ''HOST'': os.environ.get(''DB_PORT_5432_TCP_ADDR'', ''db''),
        ''PORT'': os.environ.get(''DB_PORT_5432_TCP_PORT'', ''''),
    },
}

# Redis

REDIS_PORT = 6379  
REDIS_DB = 0  
REDIS_HOST = os.environ.get(''REDIS_PORT_6379_TCP_ADDR'', ''redis'')

RABBIT_HOSTNAME = os.environ.get(''RABBIT_PORT_5672_TCP'', ''rabbit'')

if RABBIT_HOSTNAME.startswith(''tcp://''):  
    RABBIT_HOSTNAME = RABBIT_HOSTNAME.split(''//'')[1]

BROKER_URL = os.environ.get(''BROKER_URL'',  
                            '''')
if not BROKER_URL:  
    BROKER_URL = ''amqp://{user}:{password}@{hostname}/{vhost}/''.format(
        user=os.environ.get(''RABBIT_ENV_USER'', ''admin''),
        password=os.environ.get(''RABBIT_ENV_RABBITMQ_PASS'', ''mypass''),
        hostname=RABBIT_HOSTNAME,
        vhost=os.environ.get(''RABBIT_ENV_VHOST'', ''''))

# We don''t want to have dead connections stored on rabbitmq, so we have to negotiate using heartbeats
BROKER_HEARTBEAT = ''?heartbeat=30''  
if not BROKER_URL.endswith(BROKER_HEARTBEAT):  
    BROKER_URL += BROKER_HEARTBEAT

BROKER_POOL_LIMIT = 1  
BROKER_CONNECTION_TIMEOUT = 10

# Celery configuration

# configure queues, currently we have only one
CELERY_DEFAULT_QUEUE = ''default''  
CELERY_QUEUES = (  
    Queue(''default'', Exchange(''default''), routing_key=''default''),
)

# Sensible settings for celery
CELERY_ALWAYS_EAGER = False  
CELERY_ACKS_LATE = True  
CELERY_TASK_PUBLISH_RETRY = True  
CELERY_DISABLE_RATE_LIMITS = False

# By default we will ignore result
# If you want to see results and try out tasks interactively, change it to False
# Or change this setting on tasks level
CELERY_IGNORE_RESULT = True  
CELERY_SEND_TASK_ERROR_EMAILS = False  
CELERY_TASK_RESULT_EXPIRES = 600

# Set redis as celery result backend
CELERY_RESULT_BACKEND = ''redis://%s:%d/%d'' % (REDIS_HOST, REDIS_PORT, REDIS_DB)  
CELERY_REDIS_MAX_CONNECTIONS = 1

# Don''t use pickle as serializer, json is much safer
CELERY_TASK_SERIALIZER = "json"  
CELERY_ACCEPT_CONTENT = [''application/json'']

CELERYD_HIJACK_ROOT_LOGGER = False  
CELERYD_PREFETCH_MULTIPLIER = 1  
CELERYD_MAX_TASKS_PER_CHILD = 1000  

Those settings will configure the Django app so that it will discover the PostgreSQL database, Redis cache, and Celery.

Now, it''s time to connect Celery to the app. Create a file celeryconf.py and paste in this code:

myproject/celeryconf.py

import os

from celery import Celery  
from django.conf import settings

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

app = Celery(''myproject'')

CELERY_TIMEZONE = ''UTC''

app.config_from_object(''django.conf:settings'')  
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)  

That should be enough to connect Celery to our app, so the run_X scripts will work. You can read more about first steps with Django and Celery here.

Defining tasks

Celery looks for tasks inside the tasks.py file in each Django app. Usually, tasks are created either with a decorator, or by inheriting the Celery Task Class.

Here''s how you can create a task using decorator:

@app.task
def power(n):  
    """Return 2 to the n''th power"""
    return 2 ** n

And here''s how you can create a task by inheriting after the Celery Task Class:

class PowerTask(app.Task):  
    def run(self, n):
    """Return 2 to the n''th power"""
        return 2 ** n

Both are fine and good for slightly different use cases.

myproject/tasks.py

from functools import wraps

from myproject.celeryconf import app  
from .models import Job

# decorator to avoid code duplication

def update_job(fn):  
    """Decorator that will update Job with result of the function"""

    # wraps will make the name and docstring of fn available for introspection
    @wraps(fn)
    def wrapper(job_id, *args, **kwargs):
        job = Job.objects.get(id=job_id)
        job.status = ''started''
        job.save()
        try:
            # execute the function fn
            result = fn(*args, **kwargs)
            job.result = result
            job.status = ''finished''
            job.save()
        except:
            job.result = None
            job.status = ''failed''
            job.save()
    return wrapper


# two simple numerical tasks that can be computationally intensive

@app.task
@update_job
def power(n):  
    """Return 2 to the n''th power"""
    return 2 ** n


@app.task
@update_job
def fib(n):  
    """Return the n''th Fibonacci number.
    """
    if n < 0:
        raise ValueError("Fibonacci numbers are only defined for n >= 0.")
    return _fib(n)


def _fib(n):  
    if n == 0 or n == 1:
        return n
    else:
        return _fib(n - 1) + _fib(n - 2)

# mapping from names to tasks

TASK_MAPPING = {  
    ''power'': power,
    ''fibonacci'': fib
}

Building an API for scheduling tasks

If you have tasks in your system, how do you run them? In this section, you''ll create a user interface for job scheduling. In a backend application, the API will be your user interface. Let''s use the Django REST Framework for your API.

To make it as simple as possible, your app will have one model and only one ViewSet (endpoint with many HTTP methods).

Create your model, called Job, in myproject/models.py.

from django.db import models


class Job(models.Model):  
    """Class describing a computational job"""

    # currently, available types of job are:
    TYPES = (
        (''fibonacci'', ''fibonacci''),
        (''power'', ''power''),
    )

    # list of statuses that job can have
    STATUSES = (
        (''pending'', ''pending''),
        (''started'', ''started''),
        (''finished'', ''finished''),
        (''failed'', ''failed''),
    )

    type = models.CharField(choices=TYPES, max_length=20)
    status = models.CharField(choices=STATUSES, max_length=20)

    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
    argument = models.PositiveIntegerField()
    result = models.IntegerField(null=True)

    def save(self, *args, **kwargs):
        """Save model and if job is in pending state, schedule it"""
        super(Job, self).save(*args, **kwargs)
        if self.status == ''pending'':
            from .tasks import TASK_MAPPING
            task = TASK_MAPPING[self.type]
            task.delay(job_id=self.id, n=self.argument)

Then create a serializer, view, and URL configuration to access it.

myproject/serializers.py

from rest_framework import serializers

from .models import Job


class JobSerializer(serializers.HyperlinkedModelSerializer):  
    class Meta:
        model = Job

myproject/views.py

from rest_framework import mixins, viewsets

from .models import Job  
from .serializers import JobSerializer


class JobViewSet(mixins.CreateModelMixin,  
                 mixins.ListModelMixin,
                 mixins.RetrieveModelMixin,
                 viewsets.GenericViewSet):
    """
    API endpoint that allows jobs to be viewed or created.
    """
    queryset = Job.objects.all()
    serializer_class = JobSerializer

myproject/urls.py

from django.conf.urls import url, include  
from rest_framework import routers

from myproject import views


router = routers.DefaultRouter()  
# register job endpoint in the router
router.register(r''jobs'', views.JobViewSet)

# Wire up our API using automatic URL routing.
# Additionally, we include login URLs for the browsable API.
urlpatterns = [  
    url(r''^'', include(router.urls)),
    url(r''^api-auth/'', include(''rest_framework.urls'', namespace=''rest_framework''))
]

For completeness, there is also myproject/wsgi.py, defining WSGI config for the project:

import os  
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

from django.core.wsgi import get_wsgi_application  
application = get_wsgi_application()  

and manage.py

#!/usr/bin/env python
import os  
import sys

if __name__ == "__main__":  
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Leave __init__.py empty.

That''s all. Uh... lots of code. Luckily, everything is on GitHub, so you can just fork it.

Running the setup

Since everything is run from Docker Compose, make sure you have both Docker and Docker Compose installed before you try to start the app:

$ cd /path/to/myproject/where/is/docker-compose.yml
$ docker-compose build
$ docker-compose up

The last command will start five different containers, so just start using your API and have some fun with Celery in the meantime.

Accessing the API

Navigate in your browser to 127.0.0.1:8000 to browse your API and schedule some jobs.

Scale things out

Currently, we have only one instance of each container. We can get information about our group of containers with the docker-compose ps command.

$ docker-compose ps
           Name                          Command               State                                        Ports                                      
------------------------------------------------------------------------------------------------------------------------------------------------------
dockerdjangocelery_db_1       /docker-entrypoint.sh postgres   Up      0.0.0.0:5432->5432/tcp  
dockerdjangocelery_rabbit_1   /docker-entrypoint.sh rabb ...   Up      0.0.0.0:15672->15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp  
dockerdjangocelery_redis_1    /entrypoint.sh redis-server      Up      6379/tcp  
dockerdjangocelery_web_1      ./run_web.sh                     Up      0.0.0.0:8000->8000/tcp  
dockerdjangocelery_worker_1   ./run_celery.sh                  Up  

Scaling out a container with docker-compose is extremely easy. Just use the docker-compose scale command with the container name and amount:

$ docker-compose scale worker=5
Creating and starting dockerdjangocelery_worker_2 ... done  
Creating and starting dockerdjangocelery_worker_3 ... done  
Creating and starting dockerdjangocelery_worker_4 ... done  
Creating and starting dockerdjangocelery_worker_5 ... done  

Output says that docker-compose just created an additional four worker containers for us. We can double-check it with the docker-compose ps command again:

$ docker-compose ps
           Name                          Command               State                                        Ports                                      
------------------------------------------------------------------------------------------------------------------------------------------------------
dockerdjangocelery_db_1       /docker-entrypoint.sh postgres   Up      0.0.0.0:5432->5432/tcp  
dockerdjangocelery_rabbit_1   /docker-entrypoint.sh rabb ...   Up      0.0.0.0:15672->15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp  
dockerdjangocelery_redis_1    /entrypoint.sh redis-server      Up      6379/tcp  
dockerdjangocelery_web_1      ./run_web.sh                     Up      0.0.0.0:8000->8000/tcp  
dockerdjangocelery_worker_1   ./run_celery.sh                  Up  
dockerdjangocelery_worker_2   ./run_celery.sh                  Up  
dockerdjangocelery_worker_3   ./run_celery.sh                  Up  
dockerdjangocelery_worker_4   ./run_celery.sh                  Up  
dockerdjangocelery_worker_5   ./run_celery.sh                  Up  

You''ll see there five powerful Celery workers. Nice!

Summary

Congrats! You just married Django with Celery to build a distributed asynchronous computation system. I think you''ll agree it was pretty easy to build an API, and even easier to scale workers for it! However, life isn''t always so nice to us, and sometimes we have to troubleshoot.

Contribution

Original article written by Justyna Ilczuk, updated by Michał Kobus.

 ENGINEERING | DOCKER | CELERY | DJANGO | DOCKER COMPOSEShare:     

今天关于Docker Golang客户端ContainerAttach在阅读器中返回标准输入和golang docker api的分享就到这里，希望大家有所收获，若想了解更多关于A human-machine collaboration to defend against cyberattacks、Basic concepts of docker/kubernete/kata-container、centos rancher 通过本机 docker images 新增 container、Configuring and Running Django + Celery in Docker Containers等相关知识，可以在本站进行查询。

本篇文章给大家谈谈在RXJS Tap操作符中时，不会执行Console.Log，以及rxjava的操作符的知识点，同时本文还将给你拓展angular升@^6版本依赖rxjs@^6中rxjs@^6版本中操作符和静态方法报错并改正、Azure 监控诊断设置 AppServiceConsoleLogs 未显示所有 console.log 行、collections被修改; 枚举操作可能不会执行、console.dir和console.log有什么区别？等相关知识，希望对各位有所帮助，不要忘了收藏本站喔。

• 在RXJS Tap操作符中时，不会执行Console.Log（rxjava的操作符）
• angular升@^6版本依赖rxjs@^6中rxjs@^6版本中操作符和静态方法报错并改正
• Azure 监控诊断设置 AppServiceConsoleLogs 未显示所有 console.log 行
• collections被修改; 枚举操作可能不会执行
• console.dir和console.log有什么区别？

在RXJS Tap操作符中时，不会执行Console.Log（rxjava的操作符）

您可能缺少订阅。所有rxjs事情都是懒惰的，除非完成订阅，否则它们将不会运行。如果您除了tap运算符之外不需要其他方式处理结果，只需在末尾添加.subscribe()

zip(...).pipe(
  tap(...)
).subscribe();

angular升@^6版本依赖rxjs@^6中rxjs@^6版本中操作符和静态方法报错并改正

最近公司要把angular项目升级到最新版本，简单升级后系统是运行不起来的，其中就有rxjs的引入报错和rxjs一些操作符和静态方法的报错,花了一些时间排查rxjs的错误，下面就列举Observable 静态方法和操作符引用错误及解决方法和小伙伴分享一下，帮助小伙伴节省排查时间，方便小伙伴举一反三，有不正确的地方欢迎小伙伴雅正。

rxjx@^5.0.0版本Observable、静态方法、操作符的引用方法在@^6中报的错误

import {Observable} from "rxjs/Observable";
import "rxjs/add/Observable/fromEvent";
import "rxjs/add/operator/map";

ERROR in node_modules/rxjs/Observable.d.ts(1,15): error TS2307: Cannot find module ''rxjs-compat/Observable''.
src/app/common/services/http/api.ts(7,9): error TS2305: Module ''"F:/exercises/myNG/node_modules/rxjs/Observable"'' has no exported member ''Observable''.

ERROR in ./node_modules/rxjs/add/Observable/fromEvent.js
Module not found: Error: Can''t resolve ''rxjs-compat/add/observable/fromEvent'' in ''F:\exercises\myNG\node_modules\rxjs\add\Observable''

ERROR in ./node_modules/rxjs/add/Observable/fromEvent.js
Module not found: Error: Can''t resolve ''rxjs-compat/add/observable/fromEvent'' in ''F:\exercises\myNG\node_modules\rxjs\add\Observable''

rxjx@^6.0.0中的正确引用方法

import {Observable} from "rxjs/index" 和 import {Observable} from "rxjs"这两种引用都是正确的，但是在webStorm中的第二种方法会有错误信息的提示，但是可以正常编译，因为编译的时候对象会自动检索rxjs文件夹下的index.d.ts文件，但是webStorm在显示的时候不会去检索，rxjs只是一个文件夹。

import {Observable} from "rxjs/index"; //Observable
import {fromEvent} from "rxjs/index";  //静态方法
import { map } from "rxjs/operators";  //操作符

注意：正确的引用还没有完，静态方法和操作符使用方法还有变化，下面是rxjs@^5版本的实用方法及报错信息

Observable.fromEvent(someDocumen, "someEvent").startWith(0).map()

ERROR in src/.... Property ''fromEvent'' does not exist on type ''typeof Observable''.  
 
ERROR in src/....: error TS2339: Property ''startWith'' does not exist on type ''Observable<Event>''.

ERROR in src/....: error TS2339: Property ''map'' does not exist on type ''Observable<Event>''.

rxjs@^6版本的正确使用方法

静态方法直接调用就可以了，比以前版本方便不少，但是这个操作符要吐槽一下，完全没有以前的链式写法用着方便，看着舒服，理解着容易，差评！差评！！

fromEvent(someDocumen, "someEvent").pipe(
    startWith(0),
    map()
)

?catch操作符更名为catchError
?throw操作符更名为throwError

欢迎小伙伴点个赞 ??????????????????

Azure 监控诊断设置 AppServiceConsoleLogs 未显示所有 console.log 行

> console.log(/foo/);
/foo/

> console.dir(/foo/);
* /foo/
    global: false
    ignoreCase: false
    lastIndex: 0
    ...

> console.log([1,2,3])
[1, 2, 3]

> console.dir([1,2,3])
* Array[3]
    0: 1
    1: 2
    2: 3
    length: 3
    * __proto__: Array[0]
        concat: function concat() { [native code] }
        constructor: function Array() { [native code] }
        entries: function entries() { [native code] }
        ...

console.log(...)
console.dir(...)

> o = { foo: 1 }
> console.log(o)
> o.foo = 2

const defaultDate = options[0];
const defaultData = {};

export default function App() {
  const [date,setDate] = React.useState(defaultDate.value);
  const [chartData,setChartData] = useState(defaultData);

  const handleChange = (value) => {
    const date = value.value;
    setDate(date);
  };

  React.useEffect(() => {
    getDataFromDate(date).then((chartData) => {
      setChartData(chartData);
    });
  },[date]);

  return (
    <div className="card-one">
      <span className="dropdown-select">
        <Select
          options={options}
          onChange={handleChange}
        />
      </span>
      <Bar
        data={chartData}
        {...}
      />
    </div>
  );
}

const options = [
  { label: "last hour",value: "hour" },{ label: "last day",value: "day" },{ label: "last week",value: "week" },{ label: "last month",value: "month" },{ label: "last year",value: "year" }
];

const getRandomColors = (numOfBars) => {
  const letters = "0123456789ABCDEF".split("");
  let colors = [];
  for (let i = 0; i < numOfBars; i++) {
    let color = "#";
    for (let k = 0; k < 6; k++) {
      color += letters[Math.floor(Math.random() * 16)];
    }
    colors.push(color);
  }
  return colors;
};

function requestApi(date) {
  const labels = [9876,4245,2345,3452,6534];
  let result;

  switch (date) {
    case "hour":
      result = {
        labels,datasets: [
          {
            label: "Agent performance",data: [3428,8743,5748,4675,9265],backgroundColor: getRandomColors(labels.length),borderWidth: 2
          }
        ]
      };
      break;

    case "day":
      result = {
        labels,data: [3454,4555,4554,5454,4542,6543,3445,4567],borderWidth: 2
          }
        ]
      };
      break;
    default:
      break;
  }
  return Promise.resolve(result);
}

function getDataFromDate(date) {
  return requestApi(date);
}

function customTheme(theme) {
  return {
    ...theme,colors: {
      ...theme.colors,primary25: "#43425d",primary: "#3c4a64"
    }
  };
}

TypeError: Object(...) is not a function
handleChange
C:/Users/Ruben/Desktop/Projects/React/reddit2/reddit/src/features/search/SearchBar.js:14
  11 | 
  12 | 
  13 |  const handleChange = (e) => {
> 14 |    dispatch(setSearchTerm(e.target.value))
     | ^  15 |  }
  16 | 
  17 | 
View compiled
▶ 19 stack frames were collapsed.
This screen is visible only in development. It will not appear if the app crashes in production.
Open your browser’s developer console to further inspect this error.  Click the ''X'' or hit ESC to dismiss this message.

import React from "react";
import {usedispatch,useSelector} from "react-redux";
import {setSearchTerm,selectSearchTerm} from "../search/searchSlice";


export const SearchBar = () => {
  
  
  const dispatch = usedispatch();
  const term = useSelector(selectSearchTerm);


  const handleChange = (e) => {
    dispatch(setSearchTerm(e.target.value))
  }


  return(
    <div>
      <input
        id="search"
        type="text"
        value={term}
        onChange={handleChange}
        placeholder="Introduce your Topic"
        />
    </div>
  );
};

import {createSlice} from "@reduxjs/toolkit";


export const searchSlice = createSlice({
  name: ''search'',initialState: '''',reducer: {
    setSearchTerm: (state,action) => {state.search = action.payload},}
});

export const {setSearchTerm} = searchSlice.actions;

export const selectSearchTerm = (state) => state.search;

export default searchSlice.reducer;

import {configureStore} from "@reduxjs/toolkit";
import searchReducer from "../features/search/searchSlice";

export default configureStore({
  reducer: {
    search: searchReducer,},});

import React from ''react'';
import ReactDOM from ''react-dom'';
import { Provider } from "react-redux";
import ''./index.css'';
import App from ''../src/App/App'';
import reportWebVitals from ''./reportWebVitals'';
import store from "../src/App/store";

ReactDOM.render(
  <React.StrictMode>
    <Provider store={store}>
      <App />
    </Provider>  
  </React.StrictMode>,document.getElementById(''root'')
);

{
  "name": "reddit","version": "0.1.0","private": true,"dependencies": {
    "@reduxjs/toolkit": "^1.5.1","@testing-library/jest-dom": "^5.13.0","@testing-library/react": "^11.2.7","@testing-library/user-event": "^12.8.3","react": "^17.0.2","react-dom": "^17.0.2","react-redux": "^7.2.4","react-scripts": "4.0.3","web-vitals": "^1.1.2"
  },

export const searchSlice = createSlice({
  name: "search",initialState: { search: "" },// <-- object state
  reducers: {
    setSearchTerm: (state,action) => {
      state.search = action.payload; // <-- update property
    }
  }
});

export const { setSearchTerm } = searchSlice.actions;

export const selectSearchTerm = (state) => state.search.search; // <-- select property

export default searchSlice.reducer;

import React,{ useState } from 'react';

// Create your functional component:
function Example() {
  // And now you can use hooks
  // But only inside your functional component's
  // body
  const [count,setCount] = useState(0);

  return (
    <div>
      <p>You clicked {count} times</p>
      <button onClick={() => setCount(count + 1)}>
        Click me
      </button>
    </div>
  );
}