|
Server : Apache System : Linux cvar2.toservers.com 3.10.0-962.3.2.lve1.5.73.el7.x86_64 #1 SMP Wed Aug 24 21:31:23 UTC 2022 x86_64 User : njnconst ( 1116) PHP Version : 8.4.18 Disable Function : NONE Directory : /opt/alt/python37/lib/python3.7/site-packages/ssa/modules/__pycache__/ |
Upload File : |
B
��c�Q � @ s� d Z dZddlZddlZddlmZ ddlmZmZm Z ddl
m
Z
m Z ddl m
Z
ddlmZmZmZmZmZ dd lmZ ddlZdd
lmZ d d lmZ d d
l m Z m Z m Z ddl m!Z!m"Z"m#Z# ddl$m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+m,Z, yddl-m.Z/ W n e0k
�r" dZ/Y nX eG dd� d��Z1eG dd� d��Z2eG dd� d��Z3G dd� de�Z4e5e4d �d d �Z6G d d � d e�Z7e8d!k�r�e,� ej9d"ej:d#� e7� Z;e;� dS )$z'
This module contains Autotracer class
z ssa.modules� N)� defaultdict)� dataclass�field�asdict)�datetime� timedelta)� fnmatchcase)�Iterator�Optional�
NamedTuple�List�Iterable)�urlparse)�setup_database� )�Common)�iter_domains_data�iter_urls_data�get_url_durations� )�status�disabled_users�is_edition_supported)�
load_tunables)�autotracing_tunables_schema)�autotracing_stats_file)�is_xray_version_supported� sentry_initc @ sJ e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
�URLzURL data container�uri� avg_duration� max_duration�total_slow_reqs�
total_reqs�
belongs_to�belongs_to_userN)�__name__�
__module__� __qualname__�__doc__�str�__annotations__�int� r- r- �G/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/autotracer.pyr . s
r c @ s� e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < dZeed < dZeed
< dS )� RulesRejectszRejects counters containerr �non_wp� throttled�density�slowness� max_slowness�disabled�nginx� frequency� server_limit� domain_limit� no_domainN)r& r' r( r) r0 r, r+ r1 r2 r3 r4 r5 r6 r7 r8 r9 r: r- r- r- r. r/ : s
r/ c @ sD e Zd ZU dZeed< dZeed< dZeed< e e
d�Z e
ed< dS ) �Statsz AutoTracer statistics container�
rules_versionr �urls_processed�
urls_selected)�default_factory�rejectsN) r& r' r( r) r* r+ r= r, r> r r/ r@ r- r- r- r. r; J s
r; c @ s" e Zd ZU dZeed< eed< dS )�URLSz"
Representation of an URL
� domain_name�uri_pathN)r& r' r( r) r* r+ r- r- r- r. rA S s
rA )�url�returnc C s\ t | �}|jrd|j� �nd}|jr2|j� |� �nd}|j�dd�}|�d�d }t||�S )z�
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
�?� �/zwww.�:r )r �query�path�netloc�replace�splitrA )rD � fragments�qsr Z_no_www_netlocZ_no_port_netlocr- r- r. � url_split[ s rQ c s� e Zd ZdZd7� fdd� Z� fdd�Zeed�dd ��Zee e
e d
�d d ��Z e e
d
�dd�Zed�dd�Zd8e
e ed�dd�Ze e
d�dd�Zeee
d�dd�Zee e
d�d d �Zee e
d�d d �Ze
e ee
d �d!d"�Ze
e
e
d#�d$d%�Zee d�d&d'�Z d9e
e ee d(�d)d*�Z ee d�d+d,�Z e e d�d-d.�Z dd�d/d0�Z!dd�d1d2�Z"ed�d3d4�Z#ed�d5d6�Z$� Z%S ):�
AutoTracerz0
SSA autotracing module implementation.
Nc sN t � �� t�d�| _| j�dt� t� | _t� | _ d | _
|rB|nt � | _ d S )N� auto_tracerzAutoTracer enabled: %s)
�super�__init__�logging� getLogger�logger�info� __package__�listr �
tasks_list�statsr �engine)�selfr^ )� __class__r- r. rU n s
zAutoTracer.__init__c s t � s| j�d� d S t� s,| j�d� d S td krD| j�d� d S tj}|d k �r| j�d� t� �� | j�d| j � t
| j �| _ xz| �
� D ]n}| j�d|j� | j jd7 _y||j| jd� W q� tk
r� } z| j�d t|�� W d d }~X Y q�X q�W | �� d S )
Nz*AutoTracer skipped: unsupported CL editionz-AutoTracer skipped: unsupported X-Ray versionz*AutoTracer skipped: unable to import X-RayzAutoTracer startedz AutoTracer loaded config: %szStarting auto task for %sr )rD Z
tracing_countzFailed to start task: %s)r rX rY r �xray_libZstart_autotracingrT rU �debug�configr; r< r] �urls_scheduledr r> Zrequest_number� Exception�errorr* �save_iteration_stats)r_ Z
start_toolrD �e)r` r- r. �__call__x s.
(zAutoTracer.__call__)rE c C s
t dt�S )z$
Load configuration
zautotracing.json)r r r- r- r- r. � load_conf� s zAutoTracer.load_conf)rB rE c C s t dkr dS t �| �S )z;
Gets domain info for the specified domain
N)ra � domain_info)rB r- r- r. �gets_domaininfo� s zAutoTracer.gets_domaininfo)� user_namerE c C s" | j r tdkrdS t�|�jS dS )z>
Says if nginx is enabled for a specific user
NF)Z
skip_nginxra ZNginxUserCache�
is_enabled)r_ rm r- r- r. �nginx_is_enabled� s
zAutoTracer.nginx_is_enabledc C s, t dd� | jD ��}| j�d|� | j| S )z>Recalculate limit per server taking into account running tasksc S s( g | ] }|d dkr|d dkr|�qS )r �running�userz
*autotracing*r- )�.0�taskr- r- r. �
<listcomp>� s z<AutoTracer.per_server_limit_recalculated.<locals>.<listcomp>z&Number of running autotracing tasks %s)�lenr\ rX rb Zper_server_limit)r_ Z
running_countr- r- r. � per_server_limit_recalculated� s z(AutoTracer.per_server_limit_recalculated)� full_listrE c sD |dkr t �� }| j�d|� t�� t| jd� � � fdd�|D �S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)�daysc s8 g | ]0}|� d �dk r |� d �ndt� �� �kr|�qS )Z
createtimeNr )�getr, � timestamp)rr rs )�
n_days_agor- r. rt � s z1AutoTracer.excludes_old_tasks.<locals>.<listcomp>)ra r\ rX rb r �nowr Zmin_retracing_interval)r_ rw r- )r{ r. �excludes_old_tasks� s z AutoTracer.excludes_old_tasks)� current_urlrE c C s| t |�}xn| jD ]d}t |d �}|j|jko6|j|jk}t|j|j�oRt|j|j�}|s\|r| j�d||d � dS qW dS )z�
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
rD z2Skipped: URL %s was traced recently. Matched by %sTF)rQ r\ rB rC r rX rb )r_ r~ �cZ task_data�tZ direct_matchZwildcard_matchr- r- r. �exclude_thesame_urls� s z AutoTracer.exclude_thesame_urls)�url_total_reqs�domain_total_reqsrE c C s4 | j r0t�t�||��}| j�d|� || jkS dS )z-Check that URL density passes given thresholdzCalculated density %sT)r2 �npZaminZcorrcoefrX rb Zdensity_threshold)r_ r� r� Z url_densityr- r- r. �pass_by_density� s
zAutoTracer.pass_by_density)�
url_durationsrE c C s0 d| j }t�||�}| j�d||� || jkS )z�
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
�d z.Calculated %sth percentile %s for min duration)Zslow_duration_percentager� �
percentilerX rb Zslow_duration_threshold)r_ r� Zreversed_percentileZreversed_percentile_valuer- r- r. �pass_by_slowness_percentile� s
z&AutoTracer.pass_by_slowness_percentilec C s* t �|| j�}| j�d| j|� || jkS )a
The opposite to pass_by_slowness_percentile method.
The measure of "much slowness" for URL is:
at least N% of its requests must take less than X seconds.
N% -- self.max_slow_duration_percentage
X -- self.max_slow_duration_threshold
z.Calculated %sth percentile %s for max duration)r� r� Z max_slow_duration_percentagerX rb Zmax_slow_duration_threshold)r_ r� Zpercentile_valuer- r- r. � pass_by_max_slowness_percentile� s z*AutoTracer.pass_by_max_slowness_percentile)�url_throttled_reqsr� rE c C s8 |dkr dS t |�t |� d }| j�d|� || jkS )zY
Check that percent of throttled requests per URL passes given threshold
NFr� z Calculated throttled percent %s)�sumrX rb Z allowed_throttling_percentage)r_ r� r� Zthrottled_percentr- r- r. �pass_by_allowed_throttling s
z%AutoTracer.pass_by_allowed_throttling)� wp_statusrE c C s | j rt|�S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)Zonly_wp�bool)r_ r� r- r- r. �pass_by_engine s zAutoTracer.pass_by_enginec c s �x t | j�D �]�}d}| �|j�sZ| j�d� |j}| jj j |7 _ | j j
|7 _
qt t | j|j
��}�x�t| j|j
�D �]|\}}|| jkr�q~| j�d|� | j j
d7 _
| �|�d�|d �s�| j�d� | jj jd7 _q~| �|d |j��s | j�d� | jj jd7 _q~|| }| �|��sP| j�d � | jj jd7 _q~| �|��s|| j�d
� | jj jd7 _q~|dk�r�| �|j
�}|dk�r�| j�d � | jj jd7 _q~n|j }t |t t � |��t!|d �t!|d �t"|�|j
|d
�V q~W qW dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Nz Skipped by engine: non-wordpresszProcessing URL %sr r� r� z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain infoZ
url_slow_reqs)r r" r# r! r$ r% )#r r^ r� Zis_a_wordpress_domainrX rb Z urls_numberr] r@ r0 r= �dictr rB r Znon_url_fieldsr� ry r1 r� r� r2 r� r3 r� r4 rl r: rq r r, r� Zmeanr� �max) r_ � domain_data� domain_ownerZ
skipped_countZdomain_url_durationsrD �dataZ durationsrk r- r- r. � urls_computationally_filtered sb
z(AutoTracer.urls_computationally_filtered)�stats_collectedrE c c s� | � � xv| �� D ]j}|j| jkrF| j�d|j� | jj jd7 _q| � |j�rv| j�d|j� | jj j
d7 _
q|V qW dS )z�
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
z'Skipped: autotracing is disabled for %sr z Skipped: nginx is enabled for %sN) �fill_in_disabled_usersr� r% r rX rb r] r@ r5 ro r6 )r_ r� rD r- r- r. r> p s zAutoTracer.urls_selectedc c s� d}t t�}| �� }| j�d|� | �� | _| �� }x�|D ]�}| �|j �r^| j
j j d7 _ q:||k r�||j
| jk r�|d7 }||j
d7 < |V q�| j�d|j | j� | j
j jd7 _q:| j�d|j |� | j
j jd7 _q:W dS )a=
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %sr z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r, � urls_sortedrX rb r} r\ rv r� r r] r@ r7 r$ Zper_domain_limitr9 r8 )r_ Zgeneral_tasks_counterZtasks_counter_per_domainZ sorted_urlsZper_server_smart_limitrD r- r- r. rd � s.
zAutoTracer.urls_scheduledc C s, t t| �� �dd� dd�}t |dd� dd�S )zd
Sort URLs by total number of requests first
and by average duration second
c S s | j S )N)r )�ur- r- r. �<lambda>� � z(AutoTracer.urls_sorted.<locals>.<lambda>T)�key�reversec S s | j S )N)r# )r� r- r- r. r� � r� )�sortedr[ r> )r_ Z first_serier- r- r. r� � s
zAutoTracer.urls_sortedc C s t � | _ dS )z6
Fill internal list of disabled users
N)r )r_ r- r- r. r� � s z!AutoTracer.fill_in_disabled_usersc
C sp | j dk rly,ttd��}t�t| j �|� W dQ R X W n4 tk
rj } z| j�dt |�� W dd}~X Y nX dS )zI
Save collected statistics for current iteration to file
N�wz*Unable to save iteration stats to file: %s)
r] �openr �json�dumpr �OSErrorrX �warningr* )r_ �
stats_filerh r- r- r. rg � s
z AutoTracer.save_iteration_statsc
C sv yFt t��4}t�|�}t|d |d |d tf |d ��}W dQ R X W n& ttjtfk
rl t| j �}Y nX t
|�S )z@
Load statistics for latest iteration from file
r< r= r> r@ N) r� r r� �loadr; r/ r� �JSONDecodeError�KeyErrorr< r )r_ r� �_dataZ stat_datar- r- r. �load_iteration_stats� s
z AutoTracer.load_iteration_statsc C s* | � � }|�tt� d tt� �d�� |S )rG r )r Zdisabled_users_quantity)r� �updater� r ru r )r_ Z stats_loadedr- r- r. � get_stats� s
zAutoTracer.get_stats)N)N)N)&r& r' r( r) rU ri � staticmethodr� rj r* r
�objectrl r� ro r, rv r[ r} r� r� r
r� r� r� r� r r r� r> rd r r� r� rg r� r� �
__classcell__r- r- )r` r. rR i s4
W,
rR �__main__zauto_tracer_standalone.log)�filename�level)<r) rZ r� rV � collectionsr Z dataclassesr r r r r �fnmatchr �typingr r
r r r
� urllib.parser Znumpyr� Zssa.dbr �commonr Zstorager r r Z autotracingr r r Z
configurationr Zconfiguration.schemesr �internal.constantsr �internal.utilsr r Zxray.shared_libraryZshared_libraryra � ImportErrorr r/ r; rA r* rQ rR r& � basicConfig�DEBUGr� r- r- r- r. �<module>
sJ